What is multimodal AI and how do vision models work?

Multimodal AI processes more than one type of input such as text and images together. Vision-language models like GPT-4o and Claude can read screenshots, diagrams, and photos and reason about them.

AILLMMultimodal

Overview

Early LLMs understood only text. Multimodal models accept multiple types of input: images, audio, video, and documents alongside text. Vision-language models are the most common form today - you send an image (as a URL or base64-encoded bytes) with a text prompt, and the model reasons about both together. This unlocks powerful capabilities like UI analysis, document parsing, chart reading, and visual debugging.

Key Points

•GPT-4o, Claude 3.5, and Gemini 1.5 all accept images inline with text messages.
•Images cost tokens too: a 1080p image costs roughly 800-1500 tokens depending on the provider.
•Vision models can read text in images (OCR), interpret charts, describe diagrams, and compare screenshots.
•You can pass multiple images in one request for comparison tasks.
•Audio and video input is supported by Gemini 1.5 and GPT-4o Audio - not yet standard across all providers.
•Models cannot access URLs directly - you must fetch the image and encode it, or use a URL that the API server can access.

Examples

Analyse an image from a URL (OpenAI)

typescript

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image_url',
          image_url: {
            url: 'https://example.com/screenshot.png',
            detail: 'high',  // 'low' (fast, cheap) or 'high' (detailed)
          },
        },
        {
          type: 'text',
          text: 'What UI issues can you see in this screenshot? List them as bullet points.',
        },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

Send a local image as base64

typescript

import fs from 'fs';

const imageBuffer = fs.readFileSync('chart.png');
const base64Image = imageBuffer.toString('base64');

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image_url',
          image_url: {
            url: `data:image/png;base64,${base64Image}`,
          },
        },
        {
          type: 'text',
          text: 'Summarise the trend shown in this chart in one paragraph.',
        },
      ],
    },
  ],
});

Claude vision with base64 image

typescript

const response = await anthropicClient.messages.create({
  model: 'claude-sonnet-4-6',
  max_tokens: 1024,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image',
          source: {
            type: 'base64',
            media_type: 'image/png',
            data: base64Image,
          },
        },
        {
          type: 'text',
          text: 'Extract all the text you can see in this image.',
        },
      ],
    },
  ],
});

Practical use case: parse a receipt image

typescript

import { z } from 'zod';

const ReceiptSchema = z.object({
  merchant: z.string(),
  date:     z.string(),
  total:    z.number(),
  items: z.array(z.object({
    name:     z.string(),
    quantity: z.number(),
    price:    z.number(),
  })),
  currency: z.string(),
});

async function parseReceipt(imageBase64: string) {
  const response = await client.chat.completions.create({
    model: 'gpt-4o',
    temperature: 0,
    response_format: { type: 'json_object' },
    messages: [{
      role: 'user',
      content: [
        { type: 'image_url', image_url: { url: `data:image/jpeg;base64,${imageBase64}` } },
        { type: 'text', text: 'Extract all receipt data as JSON matching this schema: merchant, date, total, items (name/quantity/price), currency.' },
      ],
    }],
  });

  return ReceiptSchema.parse(JSON.parse(response.choices[0].message.content!));
}

Pro Tip

Use the low detail mode for tasks that just need a quick description or classification of an image - it is 4-6x cheaper. Switch to high detail only when the task requires reading small text or analysing fine-grained details like a UI screenshot or a hand-written form.