MBPP, short for Mostly Basic Python Problems, is a benchmark dataset designed to evaluate the code generation capabilities of large language models. It consists of 974 crowd-sourced Python programming problems that target fundamental programming skills, such as basic syntax, string manipulation, loops, and arithmetic. The benchmark is widely used in the field of Artificial intelligence to measure how well models can translate natural language problem descriptions into executable Python code.
The dataset was introduced in 2021 by a team of researchers at Google Research, including Jacob Austin, Augustus Odena, Maxwell Nye, and others. It was presented in the paper "Program Synthesis with Large Language Models," which also introduced the related APPS benchmark. MBPP was created to provide a more accessible and focused evaluation set compared to existing benchmarks, which often required solving complex algorithmic challenges or involved lengthy problem statements.
Each problem in MBPP consists of a natural language description, a function signature, and several test cases that verify the correctness of the generated code. The problems are designed to be solvable by programmers with a few months of Python experience, making the benchmark suitable for assessing basic programming competence. The dataset is split into a training set of 374 problems, a validation set of 90 problems, and a test set of 500 problems. The test set is further divided into two subsets: one for evaluating models with a single attempt (pass@1) and another for evaluating models with multiple attempts (pass@k).
Evaluation Methodology
The primary metric used for MBPP is pass@k, which measures the probability that at least one of k generated code samples passes all the provided test cases. This metric accounts for the stochastic nature of language model sampling. For example, pass@1 indicates the likelihood that a single generated solution is correct, while pass@80 measures the chance that at least one of 80 samples succeeds. The evaluation process involves generating code completions from a model given the problem description and then running the test cases against each completion.
To ensure fairness, the benchmark provides a set of canonical solutions and test cases. Models are typically evaluated in a few-shot setting, where a small number of example problems are included in the prompt to guide the model's output format. The original paper reported results for several models, including GPT-Neo, GPT-3, and a fine-tuned version of GPT-2, with pass@1 scores ranging from 3% to 37% depending on model size and training data.
Significance in AI Research
MBPP has become a standard benchmark for code generation tasks, alongside other datasets like HumanEval and APPS. Its focus on basic problems makes it particularly useful for evaluating models that are not specifically trained on code, as it tests general reasoning and programming knowledge. The benchmark has been adopted by numerous research groups and is frequently cited in papers on Machine learning and Deep learning.
One of the key advantages of MBPP is its simplicity, which allows for quick and reproducible evaluations. This has made it a popular choice for comparing the performance of different models, including those developed by OpenAI, Anthropic, and other organizations. The benchmark has also been used to study the impact of training data size, model architecture, and fine-tuning strategies on code generation ability.
Limitations and Criticisms
Despite its widespread use, MBPP has several limitations. The problems are relatively simple and may not reflect the complexity of real-world programming tasks. Additionally, the test cases are not exhaustive, so a solution that passes all tests may still contain bugs or fail on edge cases not covered by the benchmark. Some researchers have noted that the benchmark can be gamed by models that memorize solutions from the training data, as the problems are publicly available.
Another criticism is that the benchmark focuses primarily on functional correctness and does not evaluate other aspects of code quality, such as readability, efficiency, or style. This has led to calls for more comprehensive benchmarks that incorporate these factors. Nevertheless, MBPP remains a valuable tool for initial assessments of code generation models.
Applications and Extensions
MBPP has been used in various applications beyond academic research. For instance, it has been employed to evaluate the code generation capabilities of commercial AI assistants, such as GitHub Copilot and other tools built on Transformer (architecture) architectures. The benchmark has also been extended or adapted for specific purposes, such as testing multilingual code generation or evaluating models on more challenging problem sets.
In addition, MBPP has inspired the creation of similar benchmarks in other programming languages and domains. For example, the MBXP dataset extends MBPP to multiple languages, while other benchmarks focus on specific areas like data science or web development. These extensions help broaden the scope of code generation evaluation and provide more diverse challenges for AI systems.
Future Directions
As Generative AI continues to advance, benchmarks like MBPP will likely evolve to keep pace with new capabilities. Future versions may include more complex problems, incorporate real-world programming scenarios, or introduce metrics that assess code quality beyond correctness. Researchers are also exploring ways to make benchmarks more robust against data contamination and to ensure that they accurately reflect the abilities of models in practical settings.
The ongoing development of MBPP and similar benchmarks is crucial for the progress of Artificial intelligence in software engineering. By providing standardized evaluation methods, these benchmarks enable fair comparisons between models and drive improvements in code generation technology. As of 2025, MBPP remains one of the most cited benchmarks in the field, and its influence is likely to persist for years to come.