How does streaming work with LLM APIs?

Streaming delivers tokens to the client as they are generated rather than waiting for the complete response. It makes AI responses feel instant and enables progressive rendering in chat UIs.

AILLM

Overview

A non-streaming LLM API call waits for the model to finish generating the entire response before returning anything. This can take 5-30 seconds for a long answer. Streaming uses Server-Sent Events (SSE) to send each token to the client the moment it is generated. The user sees the response being typed out in real time - dramatically improving perceived performance.

Key Points

•All major LLM providers support streaming: set stream: true in OpenAI or stream: true in Anthropic.
•Each SSE event contains a delta - the new tokens generated since the last event, not the full response.
•For Next.js / React, use the Vercel AI SDK which handles streaming, state management, and UI rendering.
•Server-side you can proxy the stream: read from the LLM API stream and forward the chunks to the browser.
•Streaming does not reduce total tokens or cost - it only changes when the data arrives.
•For tool/function calling, the model streams its reasoning text but the tool call itself is a complete JSON object emitted at the end.

Examples

OpenAI streaming in Node.js

typescript

import OpenAI from 'openai';

const client = new OpenAI();

const stream = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Write a short story about a robot.' }],
  stream: true,
});

let fullResponse = '';

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content ?? '';
  process.stdout.write(delta);   // print each token as it arrives
  fullResponse += delta;
}

console.log('\n--- Complete ---');
console.log('Total chars:', fullResponse.length);

Streaming in a Next.js API route (App Router)

typescript

// app/api/chat/route.ts
import OpenAI from 'openai';
import { OpenAIStream, StreamingTextResponse } from 'ai';  // Vercel AI SDK

const client = new OpenAI();

export async function POST(req: Request) {
  const { messages } = await req.json();

  const response = await client.chat.completions.create({
    model: 'gpt-4o',
    messages,
    stream: true,
  });

  // OpenAIStream converts the OpenAI stream to a Web ReadableStream
  const stream = OpenAIStream(response);

  // StreamingTextResponse sends the stream as a streaming HTTP response
  return new StreamingTextResponse(stream);
}

Consuming the stream in a React component (Vercel AI SDK)

tsx

'use client';
import { useChat } from 'ai/react';

export default function ChatUI() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({
    api: '/api/chat',
  });

  return (
    <div>
      <div className="messages">
        {messages.map(m => (
          <div key={m.id} className={`message ${m.role}`}>
            <strong>{m.role}:</strong> {m.content}
            {/* content updates word by word as the stream arrives */}
          </div>
        ))}
        {isLoading && <div className="loading">AI is typing...</div>}
      </div>

      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} placeholder="Type a message..." />
        <button type="submit" disabled={isLoading}>Send</button>
      </form>
    </div>
  );
}

Streaming with Anthropic SDK

typescript

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

const stream = await client.messages.stream({
  model: 'claude-sonnet-4-6',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Explain recursion.' }],
});

// Option 1: listen to events
stream.on('text', (text) => process.stdout.write(text));

// Option 2: iterate over chunks
for await (const chunk of stream) {
  if (chunk.type === 'content_block_delta' && chunk.delta.type === 'text_delta') {
    process.stdout.write(chunk.delta.text);
  }
}

// Get the final complete message after streaming
const message = await stream.finalMessage();
console.log('Stop reason:', message.stop_reason);
console.log('Total tokens:', message.usage.input_tokens + message.usage.output_tokens);

Pro Tip

Always handle stream interruptions in production. Network drops or timeouts mid-stream leave users with partial responses. Implement an abort controller so users can cancel mid-stream, and show a "Connection lost - please retry" message when a stream ends unexpectedly before the stop token arrives.