Egocentric vision, also known as first-person vision, is a sub-field of computer vision that involves analyzing images and videos captured by a wearable camera, typically worn on the head or chest, which naturally approximates the visual field of the camera wearer. This perspective captures the part of the scene on which the user focuses to carry out a task, offering valuable insight into the user's activities and their context in a naturalistic setting. The forward-looking camera is often supplemented with an inward-facing camera that measures eye gaze, which helps reveal attention and better understand the user's activity and intentions.
The field has grown from early wearable devices into a distinct research area within computer vision, intersecting with Artificial intelligence and Machine learning to develop systems for activity recognition, social interaction analysis, and assistive technologies.
History
The concept of using a wearable camera to gather visual data from a first-person perspective dates back to the 1970s, when Steve Mann invented "Digital Eye Glass," a device that, when worn, effectively turned the human eye into both an electronic camera and a television display. Subsequently, wearable cameras were used for health-related applications in the context of Humanistic Intelligence and Wearable AI. Egocentric vision is best done from the point-of-eye, but may also be done via a neck-worn camera when eyeglasses would be in the way; this neck-worn variant was popularized by the Microsoft SenseCam in 2006 for experimental health research.
Interest from the computer vision community in the egocentric paradigm grew slowly entering the 2010s and has accelerated rapidly in recent years, boosted by advances in wearable technology and an increasing number of potential applications. In 2012, Kanade and Hebert described a prototypical first-person vision system composed of three basic components: a localization component to estimate the surroundings, a recognition component to identify objects and people, and an activity recognition component to provide information about the user's current activity. Together, these components provide complete situational awareness, which can be used to assist the user or a caregiver. Following this idea, early computational techniques focused on hand-related activity recognition and social interaction analysis. Given the unconstrained nature of the video and the large data volumes, temporal segmentation and summarization were among the first problems addressed.
After nearly ten years of egocentric vision research (2007-2017), the field diversified. Emerging research topics include social saliency estimation, multi-agent egocentric vision systems, privacy-preserving techniques, attention-based activity analysis, social interaction analysis, hand pose analysis, ego graphical user interfaces (EUI), understanding social dynamics and attention, revisiting robotic vision and machine vision as egocentric sensing, activity forecasting, and gaze prediction.
Technical Challenges
Wearable cameras are small, lightweight digital recording devices that capture images and video from a first-person perspective. Depending on the device, recording may be automatic, and resolution and frame rate may vary. A common distinction is made between photo-cameras (also called lifelogging cameras) and video-cameras.
Photo-cameras, such as the Narrative Clip and Microsoft SenseCam, are commonly worn on the chest and characterized by a very low frame rate (up to 2 frames per minute), allowing image capture over long periods without frequent battery recharging. They offer potential for inferring knowledge about behavior patterns, habits, or lifestyle. However, due to the low frame rate and free camera motion, temporally adjacent images often show abrupt appearance changes, making motion features unreliable.
Video-cameras, such as Google Glass and GoPro, are commonly head-mounted and capture conventional video (around 35 frames per second), enabling fine temporal detail of interactions. They offer potential for in-depth analysis of daily or special activities. However, since the camera moves with the wearer's head, estimating the wearer's global motion is more difficult, and abrupt movements can cause blurred images.
In both cases, because the camera is worn in a naturalistic setting, visual data exhibit high variability in illumination and object appearance. Moreover, the camera wearer is not visible in the image, so their actions must be inferred from the visual field, meaning important information such as pose or facial expression is unavailable.
Applications
A collection of studies published in a special theme issue of the American Journal of Preventive Medicine demonstrated the potential of lifelogs captured through wearable cameras. As a tool for understanding and tracking lifestyle behavior, lifelogs could enable prevention of noncommunicable diseases associated with unhealthy trends and risky profiles, such as obesity and depression. As a tool for re-memory cognitive training, lifelogs could help prevent cognitive and functional decline in elderly people.
More recently, egocentric cameras have been used to study human and animal cognition, human-human social interaction, human-robot interaction, and human expertise in complex tasks. Other applications include navigation and assistive technologies for the blind, monitoring and assistance of industrial workflows, and augmented-reality interfaces. The field also connects to broader Deep learning approaches for analyzing egocentric data, leveraging Neural network architectures for tasks like activity recognition and gaze prediction.
See Also
- Computer vision
- wearable-technology
- activity-recognition
- gaze-tracking