Benchmark against _current_ GPT-4, which appears to have been lobotomised...

[mccaffary]

(presumably due to model distillation or RLHF...)

[laszlovandenhoek]

I found this issue while Googling for "GPT-4 Project Euler", which are two subjects that I personally take a keen interest in :)

Re-benchmarking would be an interesting data point, especially paired with an added comparison with the "Advanced Data Analytics" plugin (formerly called "Code Interpreter", before it became generally available). I can name several reasons:

• input can be provided by uploading files (like images or large number strings)
• the underlying model (that is, GPT-4 with plugins) appears to be finetuned to leverage a Python interpreter.
• Anecdotal evidence that GPT-4+ADA is even better at problem-solving using code, which is essentially what PE is about. Indeed, perhaps the "lobotomization" of the current gpt-4-0613 with respect to the "original" gpt-4-0314 might be a result of (and perhaps, offset by) the ability to leverage plugins, which was added in that version.

For your consideration, the failures from your previous evaluation, successfully solved using ADA:

• Problem 11 (with some paraphrasing to point at the numbers marked in red):
  • https://chat.openai.com/share/7faf4574-49a3-4e3b-9bd0-016f62f864c9
• Problem 13
  • pasting the question verbatim doesn't work because the interpreter runs into its 90 second limit while repeating the numbers from the input:
    • https://chat.openai.com/share/1c02c1ad-8685-4cab-8abb-c35c3414834d
  • but providing the list as a file upload is a breeze:
    • https://chat.openai.com/share/2a4d8ba7-99d7-4001-a1a2-56e248954d1b

Also, all the problems deemed "intractable" because they were too hard to represent are easily solved by uploading the file, or simply leaving out the visual information altogether:

• Problem 15, just the text: https://chat.openai.com/share/a6d13409-9eb7-463f-8f7b-155782348dd5
• Problem 18, just the text: https://chat.openai.com/share/a8e37b34-15f4-4b59-81b5-ea50365c3265
• Problem 22, uploaded the TXT file: https://chat.openai.com/share/c16a2969-8af7-458a-96b0-d07015c773b1
• Problem 28, just the text: https://chat.openai.com/share/9a021266-b343-4dcd-aa1a-dc80711a11e0

I think it's safe to assume that GPT-4 with ADA will be able to solve everything that "plain" GPT-4 can.