Shaktidhar P 📈🌎📊 This helps. I asked:

Find the frequency and positions of the duplicate number in this series:

[325, 441, 618, 395, 233, 778, 110, 280, 384, 350, 427, 512, 997, 453, 543, 294, 516, 844, 211, 366, 230, 635, 350, 350, 948, 861, 146, 476, 132, 455, 945, 938, 881, 336, 843, 757, 952, 808, 703, 258, 734, 893, 274, 507, 277, 431, 615, 873, 108, 510, 860, 350]

Claude 3 Opus gave the right answer AND code.
GPT 4 just gave the right answer.
GPT 3.5 also got it right but hallucinated that 110 was repeated.
Claude 3 Haiku thought there were 5 repetitions, not 4.
Gemini 1.5 Pro got it plain wrong.
Llama 3 just gave the code.

How can you quickly evaluate how good an #LLM is?

I use "Gr brx vshdn Fdhvdu flskhu?

It's something #ChatGPT (GPT-4) and #Anthropic Claude 3 Opus handle, but none of the other models I've tried can understand.

(Opus in fact replies quite nicely in the same manner.)

How do YOU evaluate a new model?

Didn't know about 73 being a palindrome. Also learnt a new mathematical from the Wikipedia page: emirp (a prime that's a reverse of a prime!) 🙂

Swarna Srinivasan Quite possibly. My theory is that, like humans, LLMs try to avoid these "common" numbers when asked to pick a random one.

Overhead this comment elsewhere:

"I wonder if programmers are aware of Benford's law (tendency to start numbers with 1) and in trying to avoid it

Navin Kabra Lovely article! Made a very insightful read.

Kunal Mehta I have a feeling every LLM has explicit filters to avoid 69 😉

Leniolabs_ All thanks to you for the inspiring idea! 🙂

When picking a number between 1-100, do #LLMs pick randomly? Or pick like a human?

Leniolabs_ found #ChatGPT prefers 42. Gramener re-ran the experiment. Things have changed a bit. Now, 47 is the new favorite.

But Claude 3 Haiku latched on to 42 as its favorite. Gemini's favorite is 72.

See https://lnkd.in/gePMrhm5

They all avoid multiples of 10 (10, 20, ...), repeated digits (11, 22, ...), single digits (1, 2, ...) and prefer 7-endings (27, 37, ...). These are clearly human #biases -- avoiding regular / round numbers and seeking 7 as "random".

Strangely, they all avoid numbers ending with 1, and like 72, 73 and 56 a lot. (I don't know why. Any guesses?)

This is the coolest data visualization I've seen in a long time. It makes you think about human behaviour.

Please try and GUESS why the AirBnB occupancy rates shoot up in the red areas on Apr 7 before you read the comments!

Kripa (कृपा) Rajshekhar That's my default at the moment 🙂