ings
A theoretical physics benchmark has produced a striking result. On TheoPhysEval, a collection of 50 unpublished research-level problems, GPT-6 Astra running in Codex achieved 98% accuracy—49 correct answers out of 50—with access to literature searches, coding, and computational tools.
This does not mean that AI has solved “98% of the outstanding problems in theoretical physics.” The paper distinguishes concrete calculations from major scientific breakthroughs: the tasks mainly involve using established methods to obtain new research results, rather than resolving major open questions. Even so, the breadth of the agents’ problem-solving abilities deserves serious attention. Read the paper’s introduction.
Checking the Reference: AI Answers Prompt Experts to Revisit Their Solutions
How can researchers assess AI’s ability to tackle research problems? TheoPhysEval invited specialists across multiple fields to contribute problems whose answers were not publicly available. Its authors include researchers affiliated with ICTS, TIFR, the University of Texas at Austin, the University of Tokyo, Harvard, the University of Pennsylvania, Duke, and Google DeepMind.
The problems span quantum field theory, condensed matter physics, holography, gravitation, cosmology, and other areas. The authors estimate that a typical problem would take a leading specialist about a day or less to solve, while experts in neighboring specialties could need considerably longer.
One notable finding emerged during evaluation. Researchers initially judged Astra’s answers to several additional problems to be incorrect. Further review revealed subtle errors in some original reference solutions. Once the experts corrected those solutions, Astra’s answers were confirmed. This suggests that AI may help not only with calculations, but also with checking human-produced results. See the paper’s qualitative analysis.
From Black Holes to Quantum Physics: What Calculations Did Astra Complete?
The appendices show what these tasks actually involve. Their difficulty often lies in combining specialized knowledge, mathematical reasoning, and numerical computation. The paper records Astra as passing all four of the following tasks on the realistic track.
Black-string Page Time
This problem concerns a calculation related to black-hole information. Given a black-string background with two branes in four-dimensional anti-de Sitter space, AdS₄, the agent must compare different minimal surfaces to determine the Page time. The calculation requires handling the specified dimensions and geometry correctly, rather than directly reusing results from a different number of dimensions.
All-electron Atomic Solver
This task requires building a self-consistent atomic structure solver from scratch and calculating properties of boron and magnesium. It specifies the local density approximation, Dirac–Slater exchange, VWN5 correlation, and numerical validation. Astra passed the task’s grading criteria. That does not establish that every output matched NIST data exactly, or that the model achieved the microhartree accuracy of the reference data itself.
Axion Conversion Rate
The agent must calculate the rate at which axions convert into photons through scattering off electrons in a medium, accounting for the dielectric response and specified physical conditions. This is a theoretical calculation with defined assumptions. Passing it does not constitute the discovery of axions or a solution to the nature of dark matter.
Chern–Simons Beta Function
This problem concerns the gauge contribution to a beta function in a three-dimensional Chern–Simons matter theory, involving group-theoretic structures and renormalization calculations. Astra’s success on this highly specialized task demonstrates its ability to carry out demanding theoretical calculations.
The collection also includes problems involving boundary-layer perturbations, one-dimensional class-D fermionic systems, two-loop subdivergences, Seiberg duality, and Euclidean wormholes in type-IIB supergravity. The ability to perform calculations across such different research areas is a central feature of the results. Explore the problems in Appendix A.
Under $3 per Problem on Average: The Falling Cost of Concrete Calculations
TheoPhysEval uses two evaluation settings. The realistic track allows web searches and local tools; the isolated track disables web searches while retaining local computational tools. An isolated run therefore does not mean that the model must reason through text alone without executing code.
On the realistic track, Astra scored 98% and Opus 5.5 scored 92%. On the isolated track, both scored 92%. These results show that the agents can complete many specialized tasks without live literature searches, but the scores alone do not establish a complete understanding of the foundations of physics.
To make answers easier to verify, the evaluation converts many analytical results into numerical answers under specified parameter values. This helps reduce ambiguity from different forms of expression and makes guessing less effective. A matching numerical answer, however, does not guarantee that every intermediate step is correct.
The paper reports an average API token cost of under $3 per problem for Astra. This measures model usage, rather than the full cost of research, including problem formulation, expert verification, and infrastructure. Even with that distinction, inexpensive access to specialized calculations could change how researchers organize their work. See the evaluation methods and results.
MIT Study: Agreement with Human Conclusions Rises from 2.8% to 34.6% in 16 Months
A separate MIT study also found substantial improvement in performance on theoretical problems. Gabriel Manso, Claudio Mayrink Verdun, and Neil Thompson compared models using problems drawn from 254 recent arXiv papers across seven scientific domains.
According to MIT’s official research page, across model releases from December 2024 to April 2026, the share of problems on which models reached the same conclusion as the human authors rose from 2.8% for o1 to 34.6% for GPT-5.5. Over those 16 months, that represents an increase of 31.8 percentage points, reaching roughly 12.4 times the initial rate.
These figures support the conclusion that capabilities are improving rapidly. They do not establish the same success rate for all theoretical science problems, or guarantee that progress will continue at the same pace. Because the study and TheoPhysEval use different problem sets and evaluation methods, their percentages should not be compared directly. Read the MIT FutureTech research summary.
Beyond “Shut Up and Calculate”: How Might Scientists’ Work Change?
As AI becomes more capable at concrete calculations, scientific research still involves much more than calculation. Choosing worthwhile questions, assessing assumptions, recognizing the significance of results, and changing direction when an approach fails remain essential parts of research.
In TheoPhysEval, none of the tested agents passed an inverse problem that asked them to find an autonomous differential equation from a given curve. This illustrates that strong forward-calculation abilities do not necessarily translate into equally strong inverse-modeling abilities. Failure on one problem, however, is not sufficient to establish that AI lacks physical intuition.
The results also need to be read in light of the evaluation’s limitations: each agent completed only one full evaluation run, and the authors acknowledge that data leakage cannot be ruled out for some problems. The 98% score is therefore a reported result for this problem set and these testing conditions, rather than a guarantee of performance across all research tasks. See the paper’s introduction, methods, and discussion.
A more grounded prospect is iterative collaboration: researchers formulate questions, examine assumptions, and validate results, while AI assists with literature searches, derivations, programming, and computation. Such collaboration may accelerate research. These benchmarks, however, provide no timetable for resolving major questions such as dark matter or a grand unified theory.
