arXiv:2301.13867 · cs.LG · 2023
Mathematical Capabilities of ChatGPT
Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Julius Berner
626 citations · 25 influential citations— Semantic Scholar
TL;DR · Semantic Scholar
It is found that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a Mathematical search engine and knowledge base interface, and GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty.
Saved summary
AI-generated while a reader studied this paper. Check the original source for accuracy.
Problem: Current natural-language datasets used to benchmark language models are either very small or cover only elementary mathematics, unlike formal mathematics which possesses large databases of proofs. This lack of diverse, advanced-level natural-language datasets makes it difficult to holistically evaluate the graduate-level mathematical capabilities of models like ChatGPT and GPT-4. Consequently, professional mathematicians lack a robust framework to test whether these models can act as helpful assistants in daily research activities.
Method: The authors introduce GHOSTS and miniGHOSTS, which are the first natural-language datasets curated by working mathematics researchers to cover graduate-level mathematics and distinguish multiple dimensions of reasoning. The GHOSTS dataset consists of six subdatasets:
- Grad-Text
- Holes-in-Proofs
- Olympiad-Problem-Solving
- Symbolic-Integration
- MATH
- Search-Engine-Aspects
Using these datasets, human experts manually evaluate two iterations of ChatGPT (9-January-2023 and 30-January-2023) and GPT-4, scoring outputs from 1 (lowest) to 5 (highest) and assigning detailed error and warning codes.
Results: The 9-January-2023 and 30-January-2023 versions of ChatGPT achieve overall average ratings of only 3.20 (standard deviation of 1.23) and 3.29 (standard deviation of 1.28) respectively, failing to reach a passing grade of 3.5 on most subdatasets. GPT-4 shows a significant performance leap on the miniGHOSTS dataset with an average rating of 4.15 (standard deviation of 1.12), passing on all subdatasets except those testing graduate-level difficulty and olympiad problem-solving. While ChatGPT is best utilized as a mathematical search engine and knowledge base interface, GPT-4 handles undergraduate-level mathematics but still fails at graduate-level difficulty.
Limitations: The GHOSTS dataset is currently too small to be used for fine-tuning to improve model capabilities due to the high cost of manual annotation requiring graduate-level mathematical expertise. Additionally, the evaluated models remain inconsistently bad at advanced mathematics, are vulnerable to simple calculation errors, and almost never express uncertainty even when their output is completely wrong. Finally, prompt engineering only slightly improves overall ratings and can actually increase errors rooted in faulty logic.
Abstract
We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!
Source: https://arxiv.org/abs/2301.13867 — Thank you to arXiv for use of its open access interoperability.
Papers citing this work · Semantic Scholar
- PDE-PFN: Prior-Data Fitted Neural PDE Solver (2026) · 0 citations
- Translating AI and peer feedback in sustainability-oriented STEM teacher education: pre-service mathematics teachers’ uptake, revision, and modeling task-design quality (2026) · 0 citations
- Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation (2026) · 0 citations
- Fine-Tuning VLM for Enhancing AI's Spatial Intelligence: Understanding 3D and 2D Rotations (2026) · 0 citations
- Ascertaining Learning: Generative AI, the Proxy Problem, and Cognitive Competence in University Mathematics (2026) · 0 citations
Read the original paper, explore key passages and ask questions in AewolFlow.