https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows

https://techcrunch.com/2024/11/09/openai-reportedly-developing-new-strategies-to-deal-with-ai-improvement-slowdown/

Which gpt 4 and which gpt 3? If gpt 4o counts, it's 100 ELO better than some other versions of gpt 4, and gpt 3.5 at 1117 is less than a hundred worse than some versions of 4.

What is the answer to Capabilities(GPT4-GPT3)?

and how did that compare to Capabilities(GPT3-GPT2)?

Ah, subjective market.

I think it's hard to quantify, but I think (GPT3-GPT2) > (GPT4 - GPT-3) would be relatively non-controversial.

what if some metrics had a bigger jump but others didn't? how will this market resolve then?

@VictorLi We’ll take an average of all the top metrics and use that to start with.

If the results are still unclear or ambiguous, we’ll look at what each metric represents and I’ll use my best judgment based on what we can now do that we couldn’t do before.

Silly example: the standard metrics are all pretty middling, but also it teaches us how to speak to dolphins and turn lead into gold using nothing but Alka seltzer and chewing gum, and also decodes Linear A and proves P=NP, even if the metrics were boring in the face of undeniable new capability jumps like that which aren’t well captured by standard metrics alone I would have to call that a massive leap. (You could also express new quantifiable tests for all these new capabilities, and then test GPT4 and GPT3 against them)

Will bet when I have a balance.