What is multimodal AI? Explained with a colleague who can read, look and listen
Multimodal AI is an AI model that can take in more than one kind of information, such as text, pictures and sound, and use them together. Think of a colleague who can read, look and listen.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
In one sentence: Multimodal AI is an AI model that can take in more than one kind of information, such as text, pictures, sound or video, and use them together to answer a question or do a task.
The everyday picture
Think of two colleagues. The first can only read. If you want help with a broken machine, you must describe it in an email: the colour of the light, the words on the screen, the noise it makes.
The second colleague can read, look and listen. You send a short note, a photo of the machine and a voice memo of the noise. They look at the photo, listen to the sound, read your note, and put it all together.
Early chat assistants were like the first colleague: text in, text out. A multimodal model is the second colleague. "Modal" here means a kind of input, like words, images or sound. "Multi" means it handles several of them in one go.
You hand over words, a photo and a recording together. The model turns each into small pieces it can work with, reads them side by side, connects what they say, and writes one answer.
How it works, step by step
- You send mixed material (you hand the colleague a note, a photo and a voice memo). This could be a question with a screenshot, a scanned form or a recording. Google's Gemini documentation calls this "multimodal prompting": prompting with text, image, audio and video.
- Each input is broken into pieces (the colleague jots everything on one notepad). The model cannot use a picture as a whole. It turns it into small pieces called tokens, the same unit it uses for words. OpenAI's guide describes cutting an image into small squares that become tokens.
- The model reads it all together (the colleague looks at the notepad as one page). Words and image pieces sit side by side in the model's working space, so it can relate one to the other.
- It connects the clues (the colleague notices the error code in the photo matches your question). It can describe what it sees, answer questions about it, or summarise it.
- It writes an answer (the colleague replies). Many models answer in text only. Making new pictures is often a separate model or tool.
Try it · Everyday ↔ real
Match the colleague to the AI
Switch between the two and see how each part lines up.
- Reading a letter
- Looking at a photo
- Listening to a voicemail
- Jotting it all on one notepad
- Writing back one reply
A colleague at the next desk who can read, look and listen
| Everyday | Real |
|---|---|
| Reading a letter | Text input |
| Looking at a photo | Image input |
| Listening to a voicemail | Audio input |
| Jotting it all on one notepad | Turning inputs into tokens |
| Writing back one reply | The model's answer |
Where the picture stops
- It does not see like a person. OpenAI lists known weak spots: small or rotated text, exact positions, charts with dashed or coloured lines, and counting, where answers may be approximate.
- A good colleague says "I can't make that out". A model may describe a blurry photo confidently and wrongly. This is a kind of AI hallucination.
- Looking costs more than reading. A detailed image can use far more tokens than a sentence, and tokens are what you pay for. It also fills the model's context window faster.
Why it matters for your business
- Less retyping. Staff can send the photo, scan or screenshot instead of describing it in words.
- Faster help for customers. A customer can show the problem instead of explaining it, and the assistant can answer from what it sees.
- Watch the cost. Images and other media are billed as input, so a process that sends large files all day costs more than one that sends text. Test with your real files first.
- Keep people on high-stakes calls. Vendors warn against using these models for things like medical images, so keep a person checking important decisions.
An example
Imagine a small property maintenance company. Tenants report problems by text message, often with a photo. Today, the office manager reads each one, looks at the photo and decides who to send. With a multimodal assistant, each message and photo go in together. The assistant writes a short summary, such as "water stain on the ceiling under the bathroom, likely a leak", and suggests a category. The office manager still approves each job before anyone is sent, and checks any photo the assistant marks as unclear.
Try it · Myth or fact
Multimodal AI: myth or fact?
Decide for yourself first, then tap each statement to check it.
0 of 3 checked
Frequently asked questions
Is ChatGPT, Claude or Gemini multimodal?
Many leading assistants are. OpenAI's models accept images as well as text, and Google's Gemini also accepts audio and video. What each one supports changes often, so check the vendor's current documentation.
What is the difference between multimodal AI and a large language model?
A large language model was originally built for text. A multimodal model can also take in other kinds of input. Many of today's leading language models are multimodal.
Can a multimodal model read scanned documents?
Many can read text in images, including scans, but accuracy depends on how clear the image is. Small, rotated or faint text is a known weak spot, so test with your own documents.
Does sending a photo cost more than sending text?
Usually. Images are turned into tokens and billed as input, and larger or more detailed images use more tokens.
Related explainers
The DevDrift builds AI automation that works with your documents, photos and forms, with approvals built in: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.