A world-class expert would need about a day or less to solve one of these problems, while specialists from related disciplines would take significantly longer. Thirty-five researchers compiled 50 such problems for the TheoPhysEval dataset, which was published on October 9 on the preprint server arXiv. The tasks are drawn from ongoing research and had not been published previously. They range from fluid dynamics to electronic structure calculations for materials, quantum field theory, and cosmology. The International Centre for Theoretical Sciences in Bengaluru is leading the effort, with participants including researchers from Duke, Harvard, the University of Pennsylvania, the University of Tokyo, and KU Leuven, as well as Google DeepMind.

49 out of 50 tasks solved

Five AI agents were tested—that is, language models with access to tools such as programming environments. Using web search, OpenAI’s GPT-6 Astra solved 49 of the 50 tasks, at an average cost of less than three dollars per problem. Anthropic’s Claude Opus 5.5 solved 46, and Google’s Gemini 3.8 Flash solved 34. The open-weight models DeepSeek-V4.1-Flash and GLM 5.3 solved 27 and 21 tasks, respectively. Without internet access, both Astra and Opus solved 46 tasks each. Many tasks required formulas from specialized papers that were not provided to the agents. The authors interpret the fact that the results were hardly any worse without web search as an indication that the models have internalized large portions of the research literature. Several of Astra’s solutions were initially deemed incorrect. Upon review with the experts, it became clear that the errors lay in the experts’ model solutions.

Calculating Is Not Research

The authors believe it is possible that individual top-performing models already outperform even the best human researchers when it comes to actual calculations. However, this does not imply autonomous research. Every calculation begins with the question of which problem is worth investigating. In their experience, the agents so far lack an instinct for identifying interesting research questions, and they struggle to change direction when an approach leads nowhere. The authors draw conclusions for education. Tasks of this kind are no longer suitable as a first research project for young researchers if the goal is to produce an original result. Furthermore, more challenging problems are likely to require greater computational resources, which raises questions about access to research. They note, however, that each model was tested only once. Their conclusion alludes to the old physicist’s motto “Shut up and calculate,” which they counter with “Stop and think.”

Benchmarks at the Limit

A study conducted in September, in which physicists re-evaluated six common physics benchmarks, had already shown how quickly the models are catching up to their testing procedures. Of 250 model responses that had been rated as incorrect, only twelve contained genuine physical errors. The rest were due to flawed evaluations or reference solutions. In one of the tests, the score for OpenAI’s GPT-5.6 Sol rose from 47 to 79 percent after correction.

New Competition at the Top

On October 12, Meta unveiled Muse Wonder and Muse Wonder Full—the first models from its Superintelligence Labs—which, according to the company, compete with the leading systems from OpenAI and Anthropic. The focus is on agent tasks, programming, and a memory designed to retain relevant information over days, weeks, and months. Via Meta’s interface, Wonder costs $4 per million input tokens and $20 per million output tokens; the larger variant is reserved for subscribers to the Muse Code programming tool. The independent evaluator Artificial Analysis awarded Wonder Full 56 points on its Intelligence Index, compared to 48 for the predecessor model, Muse Spark. Meta has collected many other benchmark values itself, though they have not yet been independently verified.

Open Model from Europe

Mistral Large 4, nicknamed “Le Chonk,” comes from France and has been available as a preview version since October 6. The model has around one trillion parameters, of which approximately 49 billion are active per processed token. According to Mistral, it was trained from the ground up in Europe. Mistral plans to release the so-called weights—that is, the trained parameters—at the end of October. Companies could then run the model on their own servers. Until then, Mistral is testing it in collaboration with cybersecurity partners and government agencies. According to Artificial Analysis, Large 4 still lags significantly behind the top proprietary models. Among open models from Europe and the U.S., however, Mistral claims the top spot, particularly for cybersecurity tasks.

Physics study offers insights for industry

The physics study offers insights for industry because the tasks include fluid dynamics and material calculations—the kind of work commonly performed in development departments. Those who delegate such work to agents in the future will still need experts who can ask the right questions and verify the results.