# TREC-6

TREC-6 is a question classification benchmark with six coarse categories (ABBREVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC), used to evaluate natural language processing systems. It originated from the TREC-6 QA track in 1999.

TREC-6 is a widely used benchmark for question classification in natural language processing. It defines six coarse semantic categories into which a question is placed: ABBREVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, and NUMERIC. The dataset was introduced as part of the sixth Text REtrieval Conference (TREC-6) question answering track in 1999, and it has since become a standard testbed for evaluating classifiers that map questions to these categories.

The task is typically framed as a supervised learning problem: given a question string, a system must output one of the six labels. The original TREC-6 dataset contains 5,452 training questions and 500 test questions, with the coarse labels derived from a finer-grained hierarchy of 50 classes. The benchmark is often used alongside the TREC-10 and TREC-11 variants, but TREC-6 specifically refers to the six-class coarse setting. Researchers have used it to compare feature-based methods, such as support vector machines with lexical and syntactic features, against modern [neural network](https://www.wikiprompt.org/wiki/neural-network) approaches.

## Dataset and Task Definition

The TREC-6 dataset is derived from the TREC QA track's question collection, which includes questions from sources like the TREC-8 and TREC-9 corpora. Each question is annotated with one of six coarse types. For example, "What is the capital of France?" falls under LOCATION, while "Who wrote Hamlet?" is HUMAN. The NUMERIC category covers questions about quantities, dates, and percentages. The ENTITY category includes questions about specific objects, such as "What is the tallest building?" The ABBREVIATION category handles questions like "What does NATO stand for?" and DESCRIPTION covers questions about definitions and explanations.

The standard evaluation metric is classification accuracy on the test set. Because the classes are imbalanced (ENTITY and NUMERIC are more frequent), accuracy is still the primary reported number, but some studies also report per-class F1 scores. The benchmark is small by modern standards, but it remains useful for quick prototyping and for studying the effect of question phrasing on classification.

## Historical Context

TREC-6 was part of the TREC conference series run by the U.S. National Institute of Standards and Technology (NIST). The question answering track began in 1999 with TREC-8, but the TREC-6 label specifically refers to the sixth TREC event, which included a pilot QA task. The coarse classification scheme was formalized in later years, and the TREC-6 dataset became a common reference in academic papers. Early systems relied on hand-crafted rules and statistical classifiers, often using features like word n-grams, part-of-speech tags, and named entity recognition.

## Approaches and Performance

Classic approaches to TREC-6 include using a support vector machine with bag-of-words features, which achieved around 80-85% accuracy. More sophisticated methods incorporated syntactic parse trees and semantic role labeling. With the rise of [deep learning](https://www.wikiprompt.org/wiki/deep-learning), researchers applied convolutional neural networks and recurrent networks, often reaching 90-95% accuracy. The introduction of [transformer](https://www.wikiprompt.org/wiki/transformer)-based models, such as BERT, pushed accuracy above 97% on the test set, though the small size of the dataset means that variance across runs can be significant.

A notable technique is the use of [data augmentation](https://www.wikiprompt.org/wiki/data-augmentation) to expand the training set, since 5,452 questions is relatively limited. Some studies have also used [curriculum learning](https://www.wikiprompt.org/wiki/curriculum-learning) to order training examples by difficulty. The benchmark is often used as a sanity check for new architectures, as it is lightweight and fast to train.

## Relation to Modern Systems

While TREC-6 is a simple classification task, it remains relevant as a component of larger question answering pipelines. Modern [large language models](https://www.wikiprompt.org/wiki/large-language-model) can solve TREC-6 with near-perfect accuracy, but the benchmark is still used to evaluate the interpretability of classifiers and to test robustness to paraphrasing. The coarse categories are also used in dialogue systems and information retrieval to route queries to appropriate handlers.

The benchmark has been criticized for being too easy and for not reflecting the complexity of real-world questions, but it is still cited in papers as a baseline. The TREC-6 dataset is publicly available and can be downloaded from the NIST website or via popular machine learning libraries.

## See Also

- [Sequence-to-sequence models](https://www.wikiprompt.org/wiki/sequence-to-sequence)
- [Multi-head attention](https://www.wikiprompt.org/wiki/multi-head-attention)
- [Loss functions](https://www.wikiprompt.org/wiki/loss-functions)
- [Beam search](https://www.wikiprompt.org/wiki/beam-search)

---
Source: https://www.wikiprompt.org/wiki/trec-6
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:28:01.954063+00:00