ASDiv (A Diverse Semi-verified math word problem Dataset) is a collection of math word problems used to benchmark and train artificial intelligence systems, particularly those in natural language processing and machine learning. The dataset emphasizes diversity in problem types and language expressions, aiming to test a model's ability to generalize across different problem structures and linguistic variations. It was introduced to address limitations in earlier datasets that often contained repetitive or templated problems, which could lead to overfitting and inflated performance metrics.
The dataset consists of thousands of grade-school-level arithmetic problems, each paired with a numerical answer and a step-by-step solution. The problems cover a wide range of operations including addition, subtraction, multiplication, division, and combinations thereof, often embedded in real-world scenarios. ASDiv is notable for its semi-verified nature: problems are generated from a set of templates but then manually reviewed and adjusted to ensure diversity and correctness. This approach helps create a more challenging and realistic benchmark for evaluating math reasoning capabilities in AI models.
Design and Construction
ASDiv was constructed by researchers who recognized that existing datasets like Math23k and MAWPS often contained many similar problems, making it easier for models to memorize patterns rather than learn underlying mathematical reasoning. To mitigate this, the creators designed ASDiv with a focus on "diversity" in two dimensions: problem type and language expression. Problem types include arithmetic operations, fractions, percentages, ratios, and more. Language expression diversity is achieved by paraphrasing the same underlying math problem in multiple ways, using different vocabulary, sentence structures, and contexts.
The dataset was built using a combination of automatic generation and human verification. Initial problems were generated from a set of templates, then human annotators reviewed and edited them to ensure they were grammatically correct, mathematically sound, and sufficiently varied. This semi-verified process balances scalability with quality, resulting in a dataset that is both large enough for training and rigorous enough for evaluation.
Role in AI Research
ASDiv has become a standard benchmark in the field of AI and machine learning for evaluating math word problem solvers. It is frequently used alongside other datasets like GSM8K and MathQA to assess the reasoning abilities of large language models and transformer-based architectures. Researchers use ASDiv to measure not only accuracy but also robustness to linguistic variation, as the dataset's diversity helps reveal whether a model truly understands mathematical concepts or merely relies on surface-level pattern matching.
In recent years, ASDiv has been instrumental in highlighting the strengths and weaknesses of state-of-the-art models from organizations such as OpenAI, Anthropic, and Google DeepMind. For example, studies have shown that while large language models achieve high accuracy on ASDiv, they can still be tripped up by subtle rephrasings, indicating gaps in compositional reasoning. This has motivated research into techniques like curriculum learning and data augmentation to improve generalization.
Evaluation Metrics and Challenges
Evaluation on ASDiv typically uses exact match accuracy, where a model's predicted answer must exactly match the ground truth. However, because problems are diverse, models must handle multiple steps and avoid arithmetic errors. One challenge is that some problems require multi-step reasoning, which tests a model's ability to plan and execute a sequence of operations. Another challenge is that the dataset includes problems with extraneous information, requiring the model to identify relevant numbers and ignore distractors.
To address these challenges, researchers have developed specialized architectures and training strategies. For instance, sequence-to-sequence models with attention mechanisms have been applied, as well as encoder-decoder frameworks that generate solution expressions. More recently, large language models fine-tuned on ASDiv have shown strong performance, but the dataset remains a valuable tool for probing limitations and driving innovation.
Impact and Future Directions
ASDiv has influenced the development of subsequent datasets and benchmarks, encouraging a shift toward more diverse and challenging evaluation sets. Its emphasis on linguistic diversity has also informed data collection practices in other domains, such as reading comprehension and visual question answering. As AI systems continue to advance, ASDiv remains a relevant benchmark for assessing mathematical reasoning, a key component of general intelligence.
Future work may involve expanding ASDiv to cover more advanced topics, such as algebra and geometry, or integrating it with other modalities like diagrams. Additionally, researchers are exploring ways to use ASDiv for training models that can explain their reasoning, aligning with efforts in explainable AI. The dataset's semi-verified approach may also be adapted to other problem-solving domains, such as science or logic puzzles.