# Vision-language model

A vision-language model (VLM) is a type of multimodal AI system trained jointly on images and text to perform tasks such as image captioning, visual question answering, and grounded reasoning between the two modalities.

A vision-language model (VLM) is a [multimodal](https://www.wikiprompt.org/wiki/multimodal-ai) system trained to jointly understand images and text, enabling tasks such as image captioning, visual question answering, image-text retrieval, and grounded reasoning that connects language to specific visual content. VLMs sit within the broader category of [foundation models](https://www.wikiprompt.org/wiki/foundation-model) and are typically built on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture.

## History

Modern VLMs trace to OpenAI's [CLIP](https://www.wikiprompt.org/wiki/clip) (2021), which used contrastive learning on hundreds of millions of image-text pairs scraped from the web to align images and their captions in a shared embedding space, without requiring hand-labeled categories. CLIP's embeddings proved useful far beyond classification, powering the guidance mechanism in early [text-to-image](https://www.wikiprompt.org/wiki/text-to-image) systems and becoming a standard visual encoder for later models. Through 2021-2023, models such as Flamingo, BLIP, and LLaVA explored ways to connect a frozen or lightly-tuned vision encoder to a pretrained [large-language-model](https://www.wikiprompt.org/wiki/large-language-model), allowing the language model's reasoning and knowledge to extend to visual inputs with comparatively little additional training.

## Architecture

Most VLMs combine a vision encoder, often a [CNN](https://www.wikiprompt.org/wiki/convolutional-neural-network) or a vision transformer, with a language model backbone, connected by a projection layer that maps image features into the token embedding space the language model already understands. The combined model is then trained or fine-tuned on paired image-text data so the language model learns to attend to visual tokens the same way it attends to text tokens via the [attention-mechanism](https://www.wikiprompt.org/wiki/attention-mechanism). Some newer systems, including [Gemini](https://www.wikiprompt.org/wiki/gemini) and GPT-4o, are trained natively as unified multimodal models rather than assembled from separately pretrained pieces, blurring the line between a VLM and a general multimodal foundation model.

## Capabilities and use cases

VLMs can describe images, answer questions about charts and diagrams, read and reason about embedded text (overlapping with [OCR](https://www.wikiprompt.org/wiki/optical-character-recognition)), compare multiple images, and, when combined with [tool use](https://www.wikiprompt.org/wiki/tool-use) or [agentic](https://www.wikiprompt.org/wiki/ai-agent) loops, take actions based on what they see, as in [computer-use](https://www.wikiprompt.org/wiki/computer-use) agents that interpret screenshots to operate software. They underpin accessibility features, robotics perception, medical image triage tools, and content moderation systems that need to reason about images rather than just classify them.

## Evaluation and limitations

VLMs are evaluated on benchmarks covering visual question answering, chart and document understanding, and fine-grained recognition; performance still degrades on tasks requiring precise counting, spatial relationships, or reading small or rotated text. Like other [multimodal](https://www.wikiprompt.org/wiki/multimodal-ai) systems, VLMs can [hallucinate](https://www.wikiprompt.org/wiki/hallucination) objects or details absent from an image, a failure mode studied under "object hallucination," and their behavior can be manipulated by adversarial or misleading visual inputs, including text embedded in an image that the model treats as an instruction, a visual form of [prompt-injection](https://www.wikiprompt.org/wiki/prompt-injection).

## Notable examples

Widely used VLMs and VLM-capable systems include CLIP as a foundational encoder, LLaVA and its successors in the open research community, and multimodal releases from major labs such as [Claude](https://www.wikiprompt.org/wiki/claude), [Gemini](https://www.wikiprompt.org/wiki/gemini), and GPT-4/GPT-4o, all of which fold vision understanding into a general-purpose assistant rather than offering it as a standalone product.

---
Source: https://www.wikiprompt.org/wiki/vision-language-model
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:30:03.47588+00:00