BBH 2024 is a benchmark for evaluating large language models (LLMs) on a suite of challenging reasoning tasks. It is a further updated version of the BIG-Bench Hard (BBH) benchmark, which itself was a curated subset of the larger BIG-Bench suite. BBH 2024 was released in 2024 to address limitations of the original BBH, particularly the saturation of performance by newer models and the need for more robust evaluation protocols.
The benchmark consists of 23 tasks selected from BIG-Bench that were identified as particularly difficult for language models. These tasks cover a range of reasoning abilities, including causal judgment, date understanding, disambiguation, and logical deduction. BBH 2024 introduces revised prompts, updated answer formats, and a new scoring methodology that penalizes partially correct answers more strictly than the original BBH. The updated benchmark aims to provide a more reliable measure of model capabilities at the frontier of AI research.
Task Composition and Selection
The 23 tasks in BBH 2024 are the same as those in the original BBH, but with significant modifications. For example, the 'Causal Judgment' task now includes 100 additional counterfactual scenarios, and the 'Date Understanding' task has been expanded to cover a wider range of calendar formats. The 'Logical Deduction' task now requires models to output a sequence of reasoning steps rather than just the final answer. These changes were made to reduce the effectiveness of simple pattern matching and to encourage genuine reasoning.
Each task includes a few-shot prompt with 3 to 5 exemplars, as in the original BBH. However, BBH 2024 provides updated exemplars that reflect more recent language usage and include explanations for the correct answers. The benchmark also introduces a 'chain-of-thought' evaluation mode, where models are prompted to produce intermediate reasoning steps before the final answer, and a 'direct' mode without such prompting.
Scoring and Evaluation
BBH 2024 uses a scoring system that awards full credit only for exact matches with the ground truth answer. Partial credit is given for answers that contain the correct final answer but include extraneous text. The benchmark reports both the average accuracy across tasks and the standard deviation. For the chain-of-thought mode, the accuracy is computed on the final answer after the reasoning steps, and a separate metric, 'reasoning coherence', is introduced to assess the logical consistency of the generated steps.
In the original BBH, models such as GPT-3 achieved an average accuracy of around 40% on the tasks. By 2024, state-of-the-art models like GPT-4 and Claude 3 have surpassed 80% on the original BBH, prompting the need for a more challenging version. BBH 2024 includes a new 'hard mode' where the prompts are adversarially perturbed, and the benchmark reports results for both the standard and hard modes.
Model Performance and Saturation
Initial results on BBH 2024, released in mid-2024, show that the best-performing models, including GPT-4 Turbo and Claude 3 Opus, achieve around 70% average accuracy on the standard mode, and around 50% on the hard mode. This is a significant drop from their near-saturation on the original BBH, indicating that the updated benchmark provides a more discriminative evaluation. The benchmark has been adopted by several AI research labs, including OpenAI, Anthropic, and Google DeepMind, as a standard evaluation tool for their models.
BBH 2024 also includes a 'human baseline' study, where 100 human participants were recruited through a crowdsourcing platform. The human baseline accuracy on the standard mode is 85%, and on the hard mode is 75%, showing that the benchmark still leaves room for improvement in AI systems.
Relationship to Other Benchmarks
BBH 2024 is part of a broader ecosystem of AI benchmarks. It complements other suites such as MMLU (Massive Multitask Language Understanding) and HellaSwag, but focuses specifically on tasks that require multi-step reasoning. Unlike MMLU, which covers a wide range of knowledge domains, BBH 2024 emphasizes tasks that are easy for humans but hard for AI, as originally defined by BIG-Bench. The updated version maintains this philosophy while increasing the difficulty and robustness of the evaluation.
The benchmark is publicly available on GitHub, and the authors provide a leaderboard with results from various models. The leaderboard is updated regularly, and as of late 2024, the top entry is a model from Google DeepMind with an average accuracy of 72.3% on the standard mode.
Future Directions
The creators of BBH 2024 have indicated that they plan to release annual updates to keep pace with model improvements. They are also exploring the inclusion of tasks that require multimodal reasoning, such as interpreting charts and diagrams. The benchmark is expected to remain a key reference point for evaluating reasoning capabilities in large language models for the foreseeable future.
See Also
- BIG-Bench
- MMLU
- Chain-of-thought
- Evaluation of AI systems