Aider has had an Exercism benchmarking suite for quite some time.

Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models.

https://aider.chat/docs/benchmarks-1106.html

Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-to-apples comparison, but there doesn't seem to be a strong reason to expect qualitatively different results.

I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches.

https://github.com/AbanteAI/mentat/blob/main/tests/benchmark...

https://github.com/paul-gauthier/aider/blob/main/benchmark/p...

Edit: Not sure if the mentat authors are in this thread? After looking around a bit, there seems to be a bunch of aider code in your repo. Some attribution would be appreciated. It might even be required under aider's Apache 2.0 license?

  - C
  - C#
  - C++
  - Emacs Lisp
  - Elixir
  - Elm
  - Go
  - Java
  - Javascript
  - OCaml
  - PHP
  - Python
  - QL
  - Ruby
  - Rust
  - Typescript

The problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%).

I think the evidence from these tests are not strong enough to draw conclusions from.

The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens?

Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared)

New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion

Previous Model: gpt-4, 8192 tokens context and completion (shared)

New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion

Why would the same size of a 16K GPT-3.5 model now not allow larger completion sizes?

Why would the new GPT-4 reduce the completion tokens as well, gpt-4 can do 8192 and gpt-4-32k can do 32768 completion tokens. Now the limit is 4096.

So you would need to change the way you prompt (split the responses) to be able to get a longer response.

---

So are these new models taking the old base models of 4K tokens context and completion and changing the context to 128000 but leaving the completion the same? If they could get gpt-4 to have gpt-4-8k and gpt-4-32k, why couldn't have it been 128000 context and 32768 completion?

GPT-4 Turbo is dramatically worse at one task I often try:

Read the following passage from [new ML article]. Identify their assumptions, and tell me which mathematical operations or procedures they use depend upon these assumptions.

GPT-4: Usually correctly identifies the assumptions, and often quotes the correct mathematics in its reply.

GPT-4 Turbo: Sometimes identifies the assumptions, and is guaranteed to stop trying at that point and then give me a Wikipedia-like summary about the assumptions rather than finish the task. Further prompting will not improve its result.

> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs.

This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.

Why are all the comments here so negative... this is a good thing, turbo has less memorization but keeps the same reasoning ability. That's excellent and a relief.

I have similar conclusions so far. We have a custom data set (basically visual Q&A about web apps) and `gpt4` gets roughly 90% correct, while `gpt-4-1106-preview` only 86%. It's a little noisy (I didn't yet check out the new seeds functionality), but roughly consistent.

Since I created this dataset by hand, it can't really be memorized. I'm sure there's _similar_ data in the training set, but answering correctly still requires some reasoning-like capabilities.

Everyone who has any knowledge of machine learning knows that you don't evaluate your model by testing it on parts of its training data. The issue is, the training data for GPT-4 appears to be "everything".

I'm interested in more testing on the context side of things.

For my NLP pipelines, I batch n-articles together to process (extract fields from) in one prompt (final output is something like this {"1":[{}], "2": [{},{}]...}) in one message. Compute-wise it's inefficient but OpenAI charges by the token so it doesn't matter. It's very reliable on gpt-4 8k.

I was also pretty happy with the results on 4-turbo initially but it seems that once you go past 30k-ish tokens in context (needs way more testing), it shits itself. The indexes don't match anymore and n_final_output is different from n_articles.

Still, great model and even if the limits are lower in practice I suspect I'll get good use out of it.

Edit: With better prompting, it feels stable at n=42, ~42000 prompt tokens.

In my day job we use GPT4 quite a bit and we shifted to GPT4 Turbo today. We got a 2-5% performance increase, and quite a bit of speed increase as well.

Not to say that the parent post is incorrect, of course. Only that its not as cut and dry as a "GPT4 Turbo is distilled (read: watered down) GPT4".

This is a big problem with independent LLM testing. You need to make sure your test set isn't included in the training set which isn't easy with closed source models.

This makes me think of how hardware manufacturers optimize for benchmarks. Closed source LLMs can intentionally include likely test data in their training set to artificially inflate results. I'm not saying they are intentionally doing that now, but they could.

> Although the author OCR’ed the SAT questions and believes that they weren’t in the training data

I agree that the author of the tweet fairly underestimates the potential portion of OCR'ed contents in OpenAI's training data. In late August, Nougat[1] is released by Meta, this is an OCR model. Its performance are wild and the model is open source.

I hardly believe that OpenAI does not spend effort on getting more training from OCR content. I also hardly believes that OpenAI waits for a Meta paper to have an internal performant OCR model.

[1]: https://arxiv.org/abs/2308.13418

Very interesting and basically confirms that GPT-4 turbo is a faster but dumber model. When a task doesn't rely on memorization of the training set, it reasons similarly well to GPT-4. Where memorization is helpful, it performs worse (due to quantization-induced "memory loss").

This also makes me look at GPT-4 as a "weak reasoner with a lot of knowledge". That really aligns with my experience where it is immensely helpful and has a superhuman knowledge base but still needs handholding to solve real problems.

I've always been skeptical of benchmarking because of the memorization problem. I recently made up my own (simple) date reasoning benchmark to test this, and found that GPT-4 Turbo actually outperformed GPT-4: https://open.substack.com/pub/talcai/p/making-up-a-new-llm-b...

GPT -4 Turbo is still in preview, maybe wait until it is fully released before judging?

A little bit off topic question. When people are talking about costs with GPT, like the following link. Does the cost concern only apply to the API? If you’re using the WebUI and have a Plus account, is it always just the flat $20 amount?

https://news.ycombinator.com/item?id=38193978

Now do a programming task that requires more than 32k of context and see who’s “better”. If you don’t bench mark that you cannot get an overall pic. GitHub copilot for example could benefit big from the increased context

I think it's interesting that forcing models out of memorization don't always show a steep drop in ability.

I've definitely had instances where 4 memorized a common puzzle and failed a subtly altered variant but then got the variant after changing variable names or otherwise making it look different from what it would have memorized.

That’s why is called 4 turbo, not “4.5”. But the context length is a bigger cargo space

I wonder how often a human could guess the exercise based on just the function stub.

  return "";

This reflects my anecdotal findings in a very narrow domain: GPT-4 consistently resolves complex SQL from natural language, whereas GPT-4-Turbo is hit-and-miss, and similar to 3.5-Turbo in performance.

Is it know what exactly OpenAI does in the background when they make these turbo editions?

Seems like sacrificing some quality for large gains on speed and cost but anyone know more detail?

tldr: GPT-4 Turbo have worse score on synthetic benchmark of the first attempt because they speculate it's a smaller model, and isn't able to memorize as well the response.

Memorization often gets a bad rap as the underachiever's shortcut. However, it's a fundamental component of any learning process! Our ability to reason, solve problems, and innovate is all built upon a foundation of memorized information. In fact, it's precisely the reason humans have thrived for so long; we were able to memorize and pass down knowledge culturally long before the written word, not because we were 100 times smarter than our nearest cousins. Without memorization, be it in our brains or AI algorithms, there's no foundation to build upon for higher reasoning.

It's hard to decide for me without seeing the data. Even if you don't know the exact exercise, seeing the title and the function name/parameters is often enough for me to guess what the challenge is. I checked the public questions on exercism and almost all of those (that I spot checked) that contained the function name were extremely obvious. Knowing it's a programming challenge would also improve my guessing chances.

For example the function stubs I can find are "value_of_card(<card>)" in exercise "Black Jack", or "generate_seat_letters(<number>)" in exercise "Plane Tickets". I think I could guess those without seeing the rest of the question.

You can call it whatever you want, all I know is I used to write programs in lines of code, then blocks of code at a time, spit out by LLMs

Using GPT-4 Turbo yesterday, I feel like I'm moving to pages of code at a time now.

Taking the ideas in my head and turning them into reality is so easy now

So how can it solve novel problems? Internet does not have all combinations for every possible task with any random programming language, library or constraints. It can even solve problems with non-existing programming languages and libraries if you describe them, if that's just memorization then I don't know what it isn't.

If that's your takeaway from this then you really missed the point. The implication here is that gpt-4 to gpt-4-turbo represents a leap away from memorization and toward better reasoning with a more complete world model.

"They memorized all the problems" is not what was found here and still a wrong overcorrection.

Ok but you understand there's a body of literature that shows that LLMs don't "just" memorize

+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine.

When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first place. Then you will have to bet on all of the ambiguities of the language, cultural and meta-physical which words carry in order to hack your thing together instead of expressing yourself directly and explicitly.

Finally what is beautiful about strict code format we are so used to - it is truly the fastest and shortest path to get your thing done, in case you possess the knowledge needed.

From a black box point of view and one angle, gpt is a web filter where it will try to find you the exact thing you are looking for but from memory. Vs google you have to distill all the info into what you need

"memorize" implies they can only recite things verbatim and that's ignoring the massive leap in being able to synthesize disjoint "memories" in new ways.

even if it's not true AI or even an architecture with the potential to become AI, LLMs are already good enough to provide real world value. Obviously "super autocomplete" isn't as sexy as true AI, but still very useful

if the benchmark means replicating the experience of taking technical interviews by most people, then this is a spot on approach and serves the potential user right.

LLMs are lossy compression