EPIC-KITCHENS is a large-scale egocentric video dataset designed for research in Artificial intelligence, particularly in the fields of action recognition and action anticipation. The dataset consists of first-person (egocentric) video recordings of daily kitchen activities, captured using head-mounted cameras. It provides a rich resource for developing and evaluating algorithms that understand human interactions with objects and environments from a first-person perspective.
The dataset was introduced in 2018 by a collaboration between the University of Bristol, the University of Catania, and the University of Toronto, among others. It contains over 55 hours of video from 32 participants in their own kitchens, capturing a wide variety of cooking and preparation tasks. The videos are densely annotated with action labels, including verb and noun combinations (e.g., 'cut onion', 'pour water'), and object bounding boxes, enabling fine-grained analysis of activities.
Data Collection and Annotation
The EPIC-KITCHENS dataset was recorded using head-mounted cameras (such as GoPro) worn by participants during natural cooking sessions. The recordings took place in the participants' own homes, providing realistic and unscripted scenarios. The annotation process involved manually labeling each video frame with action segments, identifying the start and end times of actions, and assigning verb and noun labels from a predefined vocabulary. Additionally, object bounding boxes were annotated for each frame, allowing for object detection and tracking tasks.
The dataset includes a training set and a test set, with a total of 39,594 action segments and 454,714 object bounding boxes. The action labels cover 125 verb classes and 352 noun classes, resulting in a large combinatorial space of possible actions.
Benchmark Tasks and Evaluation
EPIC-KITCHENS has become a standard benchmark for egocentric vision tasks, particularly action recognition and action anticipation. Action recognition involves classifying the action being performed in a given video segment, while action anticipation requires predicting the next action before it occurs. The dataset also supports tasks such as object detection, tracking, and video retrieval.
Evaluation metrics typically include top-1 and top-5 accuracy for action recognition, and mean average precision (mAP) for anticipation tasks. The benchmark has been used in several challenges, including the EPIC-KITCHENS Action Recognition Challenge, which has been held at major computer vision conferences such as CVPR and ECCV.
Impact and Applications
The dataset has significantly advanced research in egocentric vision and has been widely adopted by the Machine learning and Deep learning communities. It has enabled the development of models that can understand human activities from a first-person perspective, which is crucial for applications in robotics, augmented reality, and assistive technologies. For example, Neural network architectures such as Transformer (architecture)-based models have been applied to EPIC-KITCHENS for action anticipation, leveraging temporal context and multimodal information.
Moreover, EPIC-KITCHENS has contributed to the study of long-term video understanding and the integration of multiple modalities, such as audio and motion. The dataset's realistic setting and dense annotations make it a valuable resource for training and evaluating models that aim to generalize to real-world scenarios.
Extensions and Related Datasets
Following the success of EPIC-KITCHENS, the same team released EPIC-KITCHENS-100, an extended version with 100 hours of video from 45 participants, including additional annotations and a larger vocabulary. Other related egocentric datasets include Ego4D, which covers a broader range of daily activities, and Charades-Ego, which focuses on egocentric action recognition in household settings. These datasets collectively push the boundaries of egocentric video understanding.
Conclusion
EPIC-KITCHENS remains a cornerstone dataset in the field of egocentric vision, providing a unique and challenging benchmark for action understanding. Its contributions have spurred numerous innovations in Artificial intelligence and continue to influence research in human-computer interaction and intelligent systems.