MathQA is a large-scale dataset introduced in 2019 by researchers from Amazon and the University of California, Santa Barbara. It contains 37,000 English multiple-choice math word problems covering topics such as general math, physics, and finance. Each problem is annotated with a linear program that specifies the sequence of operations needed to arrive at the correct answer. The dataset was created to address the limitations of earlier datasets like MathQA's predecessor, which focused on simpler arithmetic, and to support research in artificial intelligence and machine learning for mathematical reasoning.
The problems in MathQA are derived from the AQuA dataset but are re-annotated with more detailed and consistent operation programs. The annotations include a textual explanation, a formal linear program, and the final answer. The linear programs are expressed in a domain-specific language that includes operations such as addition, subtraction, multiplication, division, and comparisons. This structure allows models to learn not only the final answer but also the intermediate reasoning steps.
Dataset Structure
Each entry in MathQA consists of a problem statement, multiple-choice options, the correct answer, and a linear program. The linear program is a sequence of steps, each involving an operator and its arguments. For example, a problem about calculating simple interest might have steps that multiply the principal by the rate and then by the time. The dataset is split into training (29,837 problems), validation (4,469 problems), and test (2,985 problems) sets. The problems are categorized into 33 distinct types, such as "interest rate" or "physics" problems, which helps in analyzing model performance across different reasoning domains.
The linear programs are designed to be executable, meaning that a program interpreter can compute the answer from the given numbers. This property enables the use of program synthesis and execution techniques in model training and evaluation. The dataset also includes a set of 50,000 unlabeled problems for semi-supervised learning, though the primary benchmark uses the labeled split.
Evaluation and Benchmarks
MathQA is commonly used as a benchmark for evaluating the mathematical reasoning capabilities of AI models. The standard evaluation metric is accuracy on the test set, where a model must output the correct answer choice. Early baselines using sequence-to-sequence models and memory-augmented neural networks achieved accuracies around 30-40%. More recent approaches, including those based on Transformer (architecture) architectures and large language models, have significantly improved performance. For instance, fine-tuned versions of models like GPT-3 have reached accuracies above 80% on the test set.
The dataset is also used to evaluate the ability of models to generate interpretable reasoning steps. Since the linear programs are provided, researchers can measure not only whether the final answer is correct but also whether the predicted program matches the ground truth. This has led to the development of models that combine program generation with execution, improving both accuracy and interpretability.
Related Datasets and Tasks
MathQA is part of a broader family of math word problem datasets, including MAWPS, ASDiv, and GSM8K. Compared to these, MathQA offers a larger number of problems and more complex reasoning requirements, as many problems involve multiple steps and a wider range of mathematical operations. The dataset is often used in conjunction with other resources to train and evaluate models for natural language processing and deep learning.
The task of solving math word problems is considered a challenging test of machine comprehension and reasoning. It requires understanding the textual description, extracting relevant quantities, and applying appropriate arithmetic or algebraic operations. MathQA's annotation scheme, with explicit operation programs, provides a structured way to approach this task and has influenced subsequent dataset designs.
Limitations and Controversies
One notable limitation of MathQA is that some of the annotated programs contain errors or inconsistencies, which can mislead models during training. Researchers have identified issues such as incorrect operator usage or missing steps. Additionally, the dataset's multiple-choice format may allow models to exploit answer patterns rather than genuinely performing reasoning. Some studies have shown that models can achieve above-chance accuracy by using superficial cues, such as the length of the options or the presence of certain numbers.
Another criticism is that the problems are relatively narrow in scope, focusing on arithmetic and simple algebra, and may not fully capture the diversity of mathematical reasoning. Despite these issues, MathQA remains a widely used benchmark for evaluating and developing AI systems with mathematical problem-solving abilities.
See Also
- mathematical reasoning
- question answering
- natural language understanding
- program synthesis
- benchmark (computing)
References
- Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019). MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of NAACL-HLT.
- The MathQA dataset is available at https://math-qa.github.io/ (accessed 2025).