Multimodal

What does multimodal mean?

Multi means many, and a mode is a form of communication or input. In general English, multimodal describes anything that uses several modes: multimodal transport combines road, rail, and sea; multimodal learning mixes reading, listening, and doing. In AI, the modes are data types: text, images, audio, and video.

How multimodal AI works

The analogy: a text-only model is like a colleague you can only email. A multimodal model is one you can also hand a photo, play a recording for, or show your screen.

The precise version: each kind of input is converted into a form the model can process alongside text. Anthropic's documentation, for example, explains that Claude views an image as a grid of small patches, each treated as a visual token. Google describes its Gemini models as "built to be multimodal from the ground up," handling tasks like image captioning, visual question answering, and object detection.

Input and output are separate questions. Some models read images but only write text; others can also generate images or speech. Claude, for instance, can analyze images but cannot generate or edit them.

A real example of multimodal AI

A contractor photographs a water-stained ceiling and asks, "What could cause this, and what should I check first?" A multimodal model reads the image and the question together and replies in text. The same model could read a scanned invoice, describe a chart from a report, or turn a whiteboard photo into a to-do list.

Why multimodal matters and when to use it

Real work is not all text. Receipts, forms, screenshots, product photos, diagrams, and recorded calls all carry information. Multimodal models let one system handle them without a separate specialist tool for each. When you compare models, check which modes each supports for input and for output; our tutorial on choosing an AI model lists it as one of the deciding factors.

Related terms

• LLM: the text engine many multimodal models are built around.
• Diffusion model: the approach behind most AI image generation.
• Token: images and audio are also converted into tokens, which count toward cost and limits.
• Embedding: multimodal embeddings place text and images in the same space so you can search one with the other.

Common misconceptions about multimodal AI

• Multimodal means it can make images. Many multimodal models only understand images and reply in text.
• It sees images the way people do. Models can misread small, blurry, or rotated images and often give only approximate counts of objects.
• Images are free to send. Images are converted to tokens too, so large images add to cost and response time.

Multimodal in practice

For a voice example of more than one mode working together, see how an AI voice agent turns speech into text, reasons over it, and speaks back. Underneath, most multimodal assistants are still built around an LLM.