A vision-language model (VLM) is a multimodal system trained to jointly understand images and text, enabling tasks such as image captioning, visual question answering, image-text retrieval, and grounded reasoning that connects language to specific visual content. VLMs sit within the broader category of foundation models and are typically built on the Transformer (architecture) architecture.
History
Modern VLMs trace to OpenAI's CLIP (2021), which used contrastive learning on hundreds of millions of image-text pairs scraped from the web to align images and their captions in a shared embedding space, without requiring hand-labeled categories. CLIP's embeddings proved useful far beyond classification, powering the guidance mechanism in early Text-to-image generation systems and becoming a standard visual encoder for later models. Through 2021-2023, models such as Flamingo, BLIP, and LLaVA explored ways to connect a frozen or lightly-tuned vision encoder to a pretrained Large language model, allowing the language model's reasoning and knowledge to extend to visual inputs with comparatively little additional training.
Architecture
Most VLMs combine a vision encoder, often a CNN or a vision transformer, with a language model backbone, connected by a projection layer that maps image features into the token embedding space the language model already understands. The combined model is then trained or fine-tuned on paired image-text data so the language model learns to attend to visual tokens the same way it attends to text tokens via the Attention mechanism. Some newer systems, including Gemini and GPT-4o, are trained natively as unified multimodal models rather than assembled from separately pretrained pieces, blurring the line between a VLM and a general multimodal foundation model.
Capabilities and use cases
VLMs can describe images, answer questions about charts and diagrams, read and reason about embedded text (overlapping with OCR), compare multiple images, and, when combined with tool use or agentic loops, take actions based on what they see, as in computer-use agents that interpret screenshots to operate software. They underpin accessibility features, robotics perception, medical image triage tools, and content moderation systems that need to reason about images rather than just classify them.
Evaluation and limitations
VLMs are evaluated on benchmarks covering visual question answering, chart and document understanding, and fine-grained recognition; performance still degrades on tasks requiring precise counting, spatial relationships, or reading small or rotated text. Like other multimodal systems, VLMs can hallucinate objects or details absent from an image, a failure mode studied under "object hallucination," and their behavior can be manipulated by adversarial or misleading visual inputs, including text embedded in an image that the model treats as an instruction, a visual form of Prompt injection.
Notable examples
Widely used VLMs and VLM-capable systems include CLIP as a foundational encoder, LLaVA and its successors in the open research community, and multimodal releases from major labs such as Claude, Gemini, and GPT-4/GPT-4o, all of which fold vision understanding into a general-purpose assistant rather than offering it as a standalone product.