Streaming LLM Responses: SSE, Tokens, and Cancellation

Who this is for: frontend and full-stack developers who already call an LLM API and get a response back as one blob, and now need to make it stream — token by token, with cancel buttons, tool-call indicators, and no janky re-renders.

Streaming LLM responses is the single change that makes an AI feature feel like a product instead of a demo. A 4-second wait with a spinner reads as broken. The same 4 seconds shown as words appearing at 40 tokens/second reads as fast, even though the total latency is identical. This guide covers the transport layer (SSE vs WebSockets vs raw fetch streams), the actual wire format from OpenAI, Anthropic and Google, how to stream tool calls without flickering, cancellation that actually cancels the backend call, and the rendering tricks (markdown-while-streaming, auto-scroll, virtualization) that separate a good chat UI from a laggy one.

Why streaming matters more than raw latency

Time-to-first-token (TTFT) is the metric that determines perceived speed, not total generation time. Claude Sonnet 4.5 and GPT-5-class models typically hit TTFT of 300-800ms on a short prompt; a full 500-token response might take 6-10 seconds to fully generate. If you wait for the whole response before rendering anything, the user stares at a spinner for the full 6-10 seconds. If you stream, they see text at ~500ms and read while the rest arrives — the same total wall-clock time feels roughly 3-5x faster in user testing.

There's a second, non-cosmetic reason: streaming lets you show tool-call progress ("Searching the web...", "Reading invoice.pdf...") as it happens, which is the difference between a black box and a system the user trusts.

Transport options: SSE, WebSockets, or plain fetch

Server-Sent Events (SSE)

SSE is the default choice for LLM streaming and what OpenAI, Anthropic, and Google all use natively. It's one-directional (server to client), rides on plain HTTP, works through most corporate proxies and Cloudflare without special config, and reconnects automatically in browsers via EventSource.

event: message
data: {"type":"content_block_delta","delta":{"text":"Hello"}}

event: message
data: {"type":"content_block_delta","delta":{"text":" world"}}

event: message
data: {"type":"message_stop"}

WebSockets

Full duplex, lower per-message overhead, but you pay for it in infrastructure: sticky sessions on load balancers, more complex reconnect logic, and no benefit unless the client also needs to push data mid-stream (voice agents, collaborative editing). For a standard chat UI, WebSockets are overkill.

Fetch + ReadableStream

If you're calling the LLM API directly from an Edge function or a same-origin backend and don't need EventSource's auto-reconnect, a raw fetch with response.body.getReader() is simpler and gives you full control over parsing:

const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ messages }), signal: controller.signal });
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n\n');
  buffer = lines.pop(); // keep incomplete chunk
  for (const line of lines) {
    if (!line.startsWith('data: ')) continue;
    const payload = JSON.parse(line.slice(6));
    onDelta(payload);
  }
}

Do not put the raw LLM API key in the browser and stream directly from OpenAI/Anthropic client-side. Always proxy through your own backend so you can enforce auth, rate limits, and cost caps — see prompt caching for cost control for how to keep the proxy cheap.

The actual wire formats (2026)

Each vendor streams differently and you need to handle their event shapes, not assume a generic "delta" string.

Anthropic (Messages API, streaming): emits typed SSE events — message_start, content_block_start, content_block_delta (with text_delta or input_json_delta for tool args), content_block_stop, message_delta (carries stop_reason and usage), message_stop. Tool-call arguments stream as partial JSON strings you must accumulate and only JSON.parse once complete.

OpenAI (Responses/Chat Completions, streaming): emits chat.completion.chunk objects where choices[0].delta.content holds the incremental text, and function/tool calls stream as delta.tool_calls[].function.arguments fragments, keyed by index since multiple tool calls can interleave.

Google Gemini (streamGenerateContent): returns a stream of GenerateContentResponse objects, each with candidates[0].content.parts, and tool calls arrive as complete functionCall parts per chunk rather than fragmented JSON — simpler to parse but you get fewer, larger chunks.

// Anthropic tool-call streaming: accumulate partial JSON across chunks
let toolInputBuffer = '';
for await (const event of stream) {
  if (event.type === 'content_block_delta' && event.delta.type === 'input_json_delta') {
    toolInputBuffer += event.delta.partial_json;
  }
  if (event.type === 'content_block_stop') {
    const args = JSON.parse(toolInputBuffer); // safe only now
  }
}

If you're building anything beyond a toy, read LLM tool-calling explained and structured outputs before you design the parser — the streaming shape of a tool call is a direct consequence of how the model was told to format it.

Rendering deltas without destroying the DOM

The naive approach — re-parsing the full markdown string to HTML on every token and replacing innerHTML — is the #1 cause of janky streaming UIs. At 30-60 chunks/second, a full re-render thrashes the DOM, breaks scroll position, and on longer messages (1500+ tokens) causes visible stutter.

Practical fixes

• Buffer chunks and flush on a requestAnimationFrame or a 50-80ms interval instead of re-rendering per token. You lose zero perceived smoothness and cut render calls by 10-20x.
• Render markdown incrementally with a streaming-aware parser (react-markdown with memoized child components, or marked re-run only on the trailing incomplete block) rather than the whole message.
• Never re-mount the message component on update — key it once by message ID and mutate content via state, or React's reconciler will churn.
• For code blocks specifically, don't run syntax highlighting on every delta — highlight only after content_block_stop, and show plain monospace text while streaming.

function useStreamingBuffer(onFlush) {
  const bufRef = useRef('');
  const rafRef = useRef(null);
  const push = (chunk) => {
    bufRef.current += chunk;
    if (!rafRef.current) {
      rafRef.current = requestAnimationFrame(() => {
        onFlush(bufRef.current);
        rafRef.current = null;
      });
    }
  };
  return push;
}

Auto-scroll without fighting the user

Auto-scroll to bottom on each flush, but stop the moment the user scrolls up manually (compare scrollTop delta), and show a "jump to latest" button instead of forcing them back down. This one detail is what separates ChatGPT's feel from a hobby project's.

Cancellation that actually cancels

A cancel button that only hides the UI but leaves the backend generating tokens is a real cost leak — you're still paying per output token for a response nobody reads. Real cancellation needs to propagate three layers:

1. Client: AbortController.abort() on the fetch, which throws in your read loop — catch it and stop rendering.
2. Your backend proxy: must forward the abort to the upstream LLM call. If you're using the Anthropic/OpenAI SDK server-side, pass the same abort signal into the SDK call so the upstream HTTP connection actually closes, not just your response to the client.
3. Billing: both Anthropic and OpenAI stop charging for tokens once the connection is closed server-side (you're billed for tokens generated, not requested), so step 2 is the one that actually saves money.

// Backend: propagate abort to the upstream SDK call
app.post('/api/chat', async (req, res) => {
  const controller = new AbortController();
  req.on('close', () => controller.abort()); // client disconnected

  const stream = await anthropic.messages.stream(
    { model: 'claude-sonnet-4-5', messages, max_tokens: 1024 },
    { signal: controller.signal }
  );
  for await (const event of stream) {
    res.write(`data: ${JSON.stringify(event)}\n\n`);
  }
  res.end();
});

Test cancellation under real network conditions, not just localhost. On some hosting setups (notably serverless functions with buffering proxies in front), req.on('close') fires late or not at all — verify with your actual production infra (see Vercel vs Firebase hosting for how each handles long-lived streaming connections).

UX patterns worth copying

• Typing indicator before first token, then swap to streamed text the moment it arrives — never show both at once.
• Tool-call chips inline ("Searching...", "Running query...") that collapse into a small pill once resolved, rather than a separate log panel.
• Stop button replaces send button for the duration of the stream, and reverts on completion or error.
• Optimistic user message render — show the user's own message instantly, don't wait for a server round-trip to echo it back.
• Graceful partial-failure state: if the stream drops mid-response, keep the partial text visible and show a small "connection interrupted, retry?" affordance instead of wiping it.
• Skeleton only for structured non-text UI (e.g. a generated chart or card), never a full-page spinner once any content has begun streaming.

Framework notes

If you're on React, the Vercel AI SDK (ai package) wraps most of this — useChat gives you streaming state, cancellation, and tool-call handling out of the box, and works with Anthropic, OpenAI, and Google through a common interface. It's worth adopting even if you don't use Vercel hosting; see Vercel AI Gateway for multi-model routing if you also want provider fallback baked into the same stream. If you're rolling your own, the buffering + abort patterns above are transport-agnostic and apply the same in Vue, Svelte, or plain JS.

For non-chat use cases — a form that streams AI-generated field suggestions, or a search box with streamed autocomplete — the same SSE + buffered-render approach applies; see AI-powered site search and AI form automation for adjacent patterns.

FAQ

What's the difference between streaming and polling for LLM responses?

Polling repeatedly requests the full response until it's marked complete, which wastes requests and still shows nothing until the whole thing is done (unless the backend also chunks it, at which point you've reinvented streaming badly). SSE streaming pushes each token as it's generated over one open connection, so the client renders continuously with no extra round-trips.

Do I need WebSockets to stream LLM responses?

No. SSE (or a raw fetch ReadableStream) handles one-directional server-to-client streaming, which is all a standard chat or completion UI needs. Reach for WebSockets only if the client must also send data mid-stream, like a live voice agent or collaborative session.

Why does my streamed markdown flicker or lose formatting mid-render?

Usually because you're re-parsing the entire markdown string to HTML on every incoming token, which re-renders the whole message and can momentarily produce invalid markdown (an unclosed code fence mid-stream, for example). Buffer chunks and flush on a short interval or animation frame, and only run syntax highlighting after a code block fully closes.

Does cancelling a stream actually stop me from being billed for the rest of the tokens?

Yes, if you propagate the abort signal all the way to the upstream SDK call on your backend, not just to the client-side fetch. Both Anthropic and OpenAI bill per token actually generated and sent, so closing the upstream connection stops the meter; a client-only "cancel" that just hides the UI does not.

How do I stream tool calls without breaking the JSON?

Tool-call arguments arrive as partial JSON fragments across multiple chunks (especially with Anthropic's input_json_delta), so you must accumulate the string and only call JSON.parse once you receive the block-stop event, never on intermediate chunks.

Sources

• Anthropic Messages API streaming docs
• OpenAI streaming API reference
• Google Gemini streaming guide
• MDN: Server-Sent Events
• Vercel AI SDK documentation

---

Want a chat or copilot feature that streams properly, cancels cleanly, and doesn't flicker? Book a free 30-minute call through the contact form or message me directly on WhatsApp.