How to Stream LLM Responses in Next.js (App Router)

Nobody wants to stare at a spinner for 8 seconds while GPT-4 writes its answer. Streaming fixes this by sending tokens as they're generated, so users see output appear word by word. Here's how to wire it up end-to-end in Next.js 14 App Router.

How LLM streaming works

The API returns a ReadableStream of Server-Sent Events (SSE). Each event contains a chunk of the response. You pipe this stream from your backend to your frontend.

[User Request] → [Next.js Route Handler] → [OpenAI/Anthropic API stream] → [Frontend]
                          ↑ streams chunks back as they arrive

Step 1: Create a streaming API route

// app/api/chat/route.ts
import OpenAI from 'openai'
import { OpenAIStream, StreamingTextResponse } from 'ai' // Vercel AI SDK

const openai = new OpenAI()

export const runtime = 'edge' // edge runtime is fastest for streaming

export async function POST(req: Request) {
  const { messages } = await req.json()

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages,
    stream: true,  // ← this is the key
  })

  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}

If you'd rather not use the Vercel AI SDK, here's the raw implementation:

// Without the AI SDK — raw streaming
export async function POST(req: Request) {
  const { messages } = await req.json()

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages,
    stream: true,
  })

  const encoder = new TextEncoder()

  const stream = new ReadableStream({
    async start(controller) {
      for await (const chunk of response) {
        const text = chunk.choices[0]?.delta?.content || ''
        if (text) {
          controller.enqueue(encoder.encode(text))
        }
      }
      controller.close()
    },
  })

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
    },
  })
}

Step 2: Consume the stream on the frontend

With the Vercel AI SDK (useChat hook):

// app/chat/page.tsx
'use client'
import { useChat } from 'ai/react'

export default function ChatPage() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({
    api: '/api/chat',
  })

  return (
    <div>
      <div className="messages">
        {messages.map(m => (
          <div key={m.id} className={`message ${m.role}`}>
            {m.content}
          </div>
        ))}
        {isLoading && <div className="message assistant">▋</div>}
      </div>

      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} placeholder="Ask anything..." />
        <button type="submit" disabled={isLoading}>Send</button>
      </form>
    </div>
  )
}

Without the SDK — manual fetch + stream reading:

'use client'
import { useState } from 'react'

export default function ChatPage() {
  const [response, setResponse] = useState('')
  const [loading, setLoading] = useState(false)

  async function sendMessage(message: string) {
    setLoading(true)
    setResponse('')

    const res = await fetch('/api/chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ messages: [{ role: 'user', content: message }] }),
    })

    const reader = res.body!.getReader()
    const decoder = new TextDecoder()

    while (true) {
      const { done, value } = await reader.read()
      if (done) break
      const chunk = decoder.decode(value, { stream: true })
      setResponse(prev => prev + chunk)
    }

    setLoading(false)
  }

  return (
    <div>
      <div>{response}{loading && '▋'}</div>
      <button onClick={() => sendMessage('Hello!')}>Send</button>
    </div>
  )
}

Step 3: Stream from Anthropic (Claude)

// app/api/chat/route.ts — Anthropic version
import Anthropic from '@anthropic-ai/sdk'

const client = new Anthropic()

export async function POST(req: Request) {
  const { messages } = await req.json()

  const encoder = new TextEncoder()

  const stream = new ReadableStream({
    async start(controller) {
      const response = await client.messages.stream({
        model: 'claude-sonnet-5',
        max_tokens: 1024,
        messages,
      })

      for await (const event of response) {
        if (
          event.type === 'content_block_delta' &&
          event.delta.type === 'text_delta'
        ) {
          controller.enqueue(encoder.encode(event.delta.text))
        }
      }
      controller.close()
    },
  })

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

Common gotchas

Edge runtime required for best performance. Node.js runtime buffers responses differently. Add export const runtime = 'edge' to your route handler.

Don't forget error handling. If the stream errors mid-way, the frontend will show a partial response. Wrap the stream start in try/catch and close the controller with an error.

Abort on unmount. If the user navigates away, abort the fetch to avoid dangling requests:

useEffect(() => {
  const controller = new AbortController()
  // pass controller.signal to fetch
  return () => controller.abort()
}, [])

Shipping an AI chat interface? We build production LLM products. Get in touch.