What is multimodal AI?

Multimodal AI is a model that takes more than one kind of input — text, image, audio, video — in the same conversation and answers with all of them in view. Inside, each kind becomes a sequence of vectors, and those sequences are concatenated into a single queue. The model processes that queue exactly as it would process text, because to it there is no difference.

The vision-and-language slice is common enough to have its own name: VLM. Multimodal is the umbrella term, and it covers audio, video and generation as well. Where an input modality ends and a model capability begins is the subject of what a multimodal model is.

From the prompt writer’s side, almost nothing changes. It is still prompt engineering with one more variable. The new variable is that you can now mark up the image, crop it and point inside it, which is a subject of its own in visual prompting.

How an image becomes tokens

The step that makes all of this work was published in October 2020, and it fits in the title of the paper: an image is worth 16x16 words. Cut the image into 16-by-16 pixel squares, flatten each square into a vector, and you have a sequence. A plain transformer, with nothing about vision built into it, processes that sequence and classifies images competitively with the convolutional networks that had owned the field¹.

Do the arithmetic and the cost turns up on its own. A 1,024 by 1,024 image gives 64 by 64 squares, or 4,096 pieces. Each piece takes a slot in the sequence the way a text token would. A screenshot is a document, not a word, and the invoice reflects that.

What is still missing is the wire between that image sequence and the language model. Two ideas supplied it, in order.

The first was to train vision and text into the same space. In 2021, a team took 400 million image-caption pairs scraped off the web and trained on a task that sounds too simple to work: guess which caption belongs to which image. The result compares images against text directly, and it classified ImageNet zero-shot at the accuracy of a supervised ResNet-50 without seeing a single one of the 1.28 million training examples². That model, CLIP, became the default vision encoder of the decade.

The second was to bolt two finished models together with a piece in between. In 2022, Flamingo froze a vision model and a language model and trained only the bridge, which let it handle image and text interleaved in whatever order they arrived. With a handful of examples in the prompt, it beat models fine-tuned on thousands of times more task-specific data³.

By 2023 the recipe was cheap. The work that introduced LLaVA used a language model to generate the visual instruction data, trained a simple projection between a vision encoder and an LLM, and got multimodal assistant behaviour on an academic budget⁴. That is the architecture most open models still follow: a vision encoder, a projection layer, and the LLM you already had.

What you gain by putting them together

The gain is not that the model can see now. It is that a question can reach across both inputs.

Photograph the electricity meter and send your supplier’s tariff table as text alongside it. The photo alone gets you “I see 8,431 kWh”. The text alone gets you “depends on the reading”. Together they get you the bill. The information needed was spread across two modalities, and the answer required both at once.

That is the test that separates a real use of multimodal from a decorative one. If the question can be answered by transcribing the image first and then working in text, you did not need a multimodal model, you needed OCR. If it needs the image read in the light of the text, or the reverse, then you do.

The cases that pay for themselves tend to have the same shape. A photographed invoice against the written approval rule. An error screenshot against the documentation. A chart in a report against the quarter’s target. A floor plan against the building code. In every one of them, typing out the contents of the image would be the entire job.

The map of the subject

Five questions cover the subject, and each one is a group of articles in this guide.

See is what the model does with an image that arrives. It takes in the VLM itself, reading text inside an image, interpreting charts, multi-page documents and video.

Ask is what you do to the image before you send it. Circling a region, cropping to what matters, numbering elements, sending two versions side by side. Most of the practical gain lives here, and almost none of it is obvious.

Hear is the modality that shows up least in introductory writing and most in shipped product. Transcription with context, who spoke when, tone, long recordings. Prompting for audio has rules of its own.

Make is the reverse path: text in, image out. Generation, editing by instruction, keeping variations consistent with each other.

Fail earns a place on the map rather than a footnote, because vision failures are not random. They cluster in counting, comparison, relative position and small print, and knowing the pattern is worth more than knowing the benchmark.

Where it breaks

The gap between what these models look like they are doing and what they are actually doing is wider in vision than anywhere else, and the evidence for that is good.

On university exams, the ceiling is a long way off. MMMU collected 11.5 thousand multimodal questions from exams and college textbooks, across six disciplines, 30 subjects and 30 image types, from charts and maps to sheet music and chemical structures. In the evaluation published in November 2023, the best models of the day landed at 56% and 59% accuracy⁵. These are questions an undergraduate answers.

On basic visual tasks, the result is worse than on the exams. A July 2024 paper assembled BlindTest, seven trivial tasks: do two circles overlap, how many times do two lines cross, which letter in a word is circled, how many circles are in an Olympic-style logo. Four frontier models averaged 58.07%, and the best of them reached 77.84%, against the 100% expected of a person⁶. The models tested were the mid-2024 ones and the numbers have certainly moved since; the shape of the error is the part that keeps.

The same study isolated where it breaks, and that is the useful part. When the shapes were spaced well apart, accuracy went to near 100%. The trouble showed up when elements overlapped or sat close together. And linear probes indicated the vision encoder held enough information to solve the tasks: what failed was turning that information into words⁶.

The blindness has a known cause. A January 2024 paper found pairs of images that CLIP treats as similar despite being plainly different, and built a benchmark out of them. Frontier models, commercial ones included, got direct questions about nine basic visual patterns wrong, and defended the wrong answers with invented explanations. The correlation between what confuses CLIP and what confuses the multimodal model was clear⁷. If the encoder cannot tell two things apart, the language model sitting on top of it has nothing to tell them apart with.

Three practical consequences follow. Do not ask for an exact count in a dense image. Do not trust a judgement of relative position between nearby elements. And treat the explanation with particular suspicion: when perception goes wrong, the model writes a coherent justification for whatever it thinks it saw.

What it costs

Order-of-magnitude numbers, with the assumptions in plain sight, because prices move and mechanisms do not.

An image’s token count comes out of its resolution. By the patch arithmetic, 1,024 by 1,024 pixels gives 4,096 pieces; in practice providers pool and compress, and an image that size usually bills as a few hundred to a few thousand input tokens. The order of magnitude is the point: an image costs like a page of text, not like a sentence.

The corollary is the most direct cost lever available here. Downscale the image before you send it, down to the limit where what matters is still legible. A 3,840 by 2,160 screenshot resized to 1,280 wide usually keeps everything that counts and cuts the bill by a multiple rather than a fraction.

The second corollary is to crop instead of downscale when the target is small. If the question is about one field on a form, send the field. That fixes cost and accuracy in the same move, because it takes away the crowded-elements case the model is worst at.

Video is the one that surprises the invoice. A video is a sequence of frames, and even sampling one frame per second, a three-minute clip is 180 images. The question to ask before sending video is always the same: how many frames actually answer this?

Where to start

1. Check that the question crosses the modalities. If transcribing the image settles it, use OCR and work in text. Cheaper, faster and far easier to debug.
2. Crop before you send. The region that matters, at the size where it is legible. Cropping improves accuracy and cost in a single action.
3. Ask for what is visible before you ask for the conclusion. “List the fields you can read” before “approve or reject”. You get to see where the error came from.
4. Do not ask for exact counts or relative position in a dense image. It is the best-documented failure mode. If you have to count, crop and count in parts.
5. Treat the explanation as suspect. Bad perception arrives with a convincing justification attached. The explanation is not evidence that it saw anything.
6. Build a test set out of your own images. Twenty real ones, with the right answer written down. A public benchmark says little about a creased invoice photographed under your lighting.
7. Measure resolution as a parameter. Run the same set at two or three resolutions. There is usually a point where accuracy stops climbing and cost carries on.

Step 7 is the one almost nobody does and the one that saves the most. The resolution your code is sending today was chosen by the user’s phone, not by you.

Footnotes

1. Dosovitskiy et al. (2020) showed that a pure transformer applied directly to sequences of image patches reaches results competitive with convolutional networks on classification, using fewer training resources. ↩

2. Radford et al. (2021) trained on 400 million image-caption pairs to predict which caption belongs to which image, and obtained zero-shot transfer matching the accuracy of ResNet-50 on ImageNet without using any of its 1.28 million labelled examples. ↩

3. Alayrac et al. (2022) wired together an already-trained vision model and language model, with support for interleaved image and text, and beat models fine-tuned on thousands of times more task data using only a few examples in the prompt. ↩

4. Liu et al. (2023) generated visual instruction data with a language model and trained the connection between a vision encoder and an LLM, establishing the encoder-plus-projection recipe that open models follow. ↩

5. Yue et al. (2023) assembled 11.5 thousand college-level multimodal questions across six disciplines and 30 image types; the best models evaluated in the paper reached 56% and 59% accuracy. ↩

6. Rahmanzadehgervi et al. (2024) measured 58.07% average accuracy across four frontier models on seven trivial visual tasks, with 77.84% for the best of them; accuracy rose to near 100% when the shapes were spaced apart, and linear probes indicated the information was present in the vision encoder and lost in the translation to text. ↩ ↩²

7. Tong et al. (2024) identified image pairs that CLIP treats as similar despite obvious differences, showed that frontier multimodal models get direct questions about nine basic visual patterns wrong, and produce invented explanations for the errors. ↩