Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR), is the application of Computer vision techniques to the image retrieval problem: the task of searching for digital images in large databases. The defining characteristic of CBIR is that the search analyzes the contents of the image itself, such as colors, shapes, textures, or other derivable information, rather than metadata like keywords, tags, or descriptions. This approach directly opposes traditional concept-based methods, which depend on human annotation quality and completeness, a dependency that becomes problematic for very large or automatically generated image collections.
CBIR systems employ computational methods from fields such as statistics, pattern recognition, signal processing, and Computer vision. The field has evolved significantly since its origins in the early 1990s, driven by the growth of digital image repositories and the impracticality of manual annotation at scale. While early systems relied on basic visual feature extraction, modern implementations increasingly leverage Machine learning and Deep learning techniques to improve retrieval accuracy and handle more complex queries.
Historical Development
The term "content-based image retrieval" was first used in 1992 by Japanese Electrotechnical Laboratory engineer Toshikazu Kato, who described experiments in automatically retrieving images from a database based on their colors and shapes. This marked a shift from prior approaches that relied solely on textual descriptions. The earliest commercial CBIR system was developed by IBM and called QBIC (Query By Image Content). While the storage of multiple images as part of a single entity predated the term BLOB (Binary Large OBject), the ability to search fully by content, rather than description, had to await IBM's QBIC system. Later developments included network- and graph-based approaches that offered simple and attractive alternatives to existing methods.
Comparison with Metadata-Based Searching
Traditional image search requires humans to manually annotate images with keywords or metadata in a database, a time-consuming process that may not capture the criteria desired for describing an image effectively. The evaluation of keyword-based search effectiveness is subjective and has not been well-defined, and CBIR systems face similar challenges in defining success. Keywords also limit the scope of queries to a predetermined set of criteria, and such annotations are less reliable than analyzing the actual image content. CBIR becomes particularly valuable for very large databases or for images generated automatically, such as those from surveillance cameras, where manual annotation is impractical. Systems categorizing images into semantic classes (for example, "cat" as a subclass of "animal") can reduce miscategorization issues but require more user effort to locate images that might belong to a broader category only.
Query Techniques
CBIR systems support various query methods, tailored to different user needs. The most common is Query By Example (QBE), where the user provides an example image, either by supplying a preexisting image, choosing one from a random set, or drawing a rough approximation using color blobs or general shapes. Results should share common elements with the provided example, and this technique removes difficulties associated with describing images in words.
Semantic retrieval starts with requests like "find pictures of Abraham Lincoln," which is challenging for computers because the subject may not always appear in the same pose or orientation. Therefore, many CBIR systems rely on lower-level features like texture, color, and shape, either combined with interfaces allowing easier input of criteria or with databases pre-trained to match specific features such as faces, fingerprints, or shapes. In general, image retrieval of higher-level concepts requires human feedback.
Other query methods include browsing for example images, navigating customized or hierarchical categories, querying by image region rather than the entire image, querying by multiple example images, querying by visual sketch, direct specification of image features, and multimodal queries combining touch, voice, and other inputs.
Relevance Feedback and Machine Learning
An important aspect of CBIR success is understanding user intent. Relevance feedback allows users to progressively refine search results by marking images as "relevant," "not relevant," or "neutral" to the query, after which the system repeats the search incorporating this new information. This interactive approach, along with other human-centered design elements, has become a key focus in CBIR development. The field has also seen increasing use of Machine learning and iterative techniques to improve retrieval performance. Initial CBIR systems searched databases based on color, texture, and shape properties; after these systems were developed, the need for user-friendly interfaces became apparent, leading to efforts that included query methods supporting descriptive semantics, user feedback, and systems capable of understanding user satisfaction levels.
Technical Progress and Challenges
Interest in CBIR has grown due to the limitations of metadata-based systems and the wide range of potential applications for efficient image retrieval. Textual information is easily searchable with existing technology, but manual description of every image is impractical for very large or automatically generated collections, and descriptions using different synonyms can cause images to be missed. Various standards have been developed to categorize images, but all face scaling and miscategorization issues. Many CBIR systems have been developed, but as of 2006, the problem of retrieving images based on pixel content remained largely unsolved. The most common method for comparing two images (typically an example image and an image from the database) involves computing a distance measure based on extracted visual features, with results ranked accordingly. Modern systems are increasingly incorporating Deep learning and Neural network architectures to improve feature extraction and semantic understanding, though full semantic retrieval at human levels remains an open research challenge.