I virtually shipped a foul AI integration as soon as. The characteristic labored technically: the mannequin answered questions accurately, the UI seemed clear, and everybody nodded alongside in our inner demo. Then we put actual customers on it, and inside 48 hours a product supervisor Slack’d me asking why 60% of periods had been ending with no response obtained.
The issue wasn’t the API or the server. It was eight seconds of silence.
Customers despatched a message, noticed nothing occur, assumed the request had failed, and closed the tab. The response would arrive shortly after, to no person. We had constructed a useful AI characteristic that felt utterly damaged, and the repair wasn’t an architectural overhaul or a efficiency optimization. It was streaming: begin sending tokens the second the mannequin produces them, as a substitute of batching the total response and delivering it without delay.
I’ve seen this error in codebases throughout a number of corporations now. The non-streaming model is less complicated to jot down, assessments go, and no person notices in improvement as a result of builders anticipate issues, whereas actual customers normally don’t.
This tutorial builds the entire streaming implementation: a Node.js backend utilizing Server-Despatched Occasions to push tokens to the browser as they arrive, with session administration that provides the mannequin reminiscence throughout turns, and correct abort dealing with so you aren’t paying for tokens no person reads.
The whole code is at github.com/ziaongit/nodejs-openai-streaming.
No OpenAI key? I examined this whole article utilizing Groq as a substitute. It’s free, it speaks the identical OpenAI SDK format, and getting a key takes about two minutes. The repo has a
USE_GROQ=trueflag already wired in. Copy.env.instanceto.env, paste your Groq key, and you might be working. I usedllama-3.3-70b-versatile. The output is sweet sufficient that you wouldn’t discover the distinction in a dev session.
Stipulations
- Node.js v18.11 or later
- An OpenAI API key
- Strong familiarity with Categorical and async/await
The Structure Earlier than Any Code
Earlier than we write any code, it helps to see what is definitely occurring throughout the three layers. Debugging streaming points with out this image will get painful quick.
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ 1. POST /session ─────────────────────► receives sessionId │
│ 2. POST /chat { message, sessionId } │
│ 3. response.physique.getReader() ──► ReadableStream │
│ 4. decode chunks ──► parse SSE occasions ──► append tokens to UI │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTP (textual content/event-stream)
▼
┌──────────────────────────────────────────────────────────────────┐
│ Categorical Server │
│ │
│ periods Map ── { sessionId: { messages: […] } } │
│ │
│ POST /chat │
│ ├─ load session historical past │
│ ├─ push person message │
│ ├─ set SSE headers + flushHeaders() │
│ ├─ openai.create({ stream: true, sign: controller.sign }) │
│ ├─ for await chunk → res.write(`information: ${token}nn`) │
│ ├─ accumulate fullResponse → push to session │
│ └─ ship { performed: true } → res.finish() │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTPS chunked switch encoding
▼
┌──────────────────────────────────────────────────────────────────┐
│ OpenAI API │
│ gpt-4o-mini · stream: true │
└──────────────────────────────────────────────────────────────────┘
The session retailer lives on the server. Each time the person sends a message, that full dialog historical past goes to OpenAI contained in the messages array, which is how the mannequin is aware of what was mentioned earlier. The API itself remembers nothing between calls, so that you carry the state.
The mechanism is less complicated than it sounds: open an everyday HTTP response and by no means shut it. The server retains writing chunks into the socket as tokens arrive, and when the mannequin finishes you name res.finish() and the connection drops.
Challenge Setup
mkdir nodejs-openai-streaming
cd nodejs-openai-streaming
npm init -y
npm set up categorical openai cors dotenv
mkdir public
Add `"kind": "module"` to `bundle.json`:
{
"title": "nodejs-openai-streaming",
"model": "1.0.0",
"kind": "module",
"scripts": {
"begin": "node server.js",
"dev": "node --watch server.js"
},
"dependencies": {
"cors": "^2.8.6",
"dotenv": "^16.6.1",
"categorical": "^4.22.2",
"openai": "^4.104.0"
}
}
Create .env:
Try our hands-on, sensible information to studying Git, with best-practices, industry-accepted requirements, and included cheat sheet. Cease Googling Git instructions and truly be taught it!
OPENAI_API_KEY=your_api_key_here
PORT=3000
Challenge construction:
nodejs-openai-streaming/
├── server.js
├── public/
│ └── index.html
├── .env
├── .env.instance
├── .gitignore
└── bundle.json
Why SSE and Not WebSockets
Folks ask about WebSockets each time, normally as a result of they sound extra succesful and subsequently extra right.
For AI chat, they normally will not be. Responses journey a method, server to shopper, and SSE is constructed for precisely that. It runs over plain HTTP with no protocol improve and no particular reverse proxy configuration, and it really works by way of HTTP/2 with out touching a config file. I’ve shipped SSE to manufacturing a number of occasions and by no means as soon as frightened about browser help.
WebSockets make sense when the shopper must push information again on the similar frequency the server pushes it: gaming, shared whiteboards, dwell bidding. In an AI chat, the person sorts one message and sits there studying, so SSE suits the site visitors sample higher.
One real limitation of SSE price realizing: the browser’s built-in EventSource API solely helps GET requests. Since it’s worthwhile to submit a message physique, you utilize fetch with a ReadableStream reader as a substitute. The SSE wire protocol is equivalent; you might be simply studying the stream manually somewhat than by way of the browser’s occasion wrapper.
Session Administration
The primary chatbot I constructed on high of this API had a bug I couldn’t determine for 2 days. Customers would ask a follow-up query and the mannequin would reply prefer it was listening to from them for the primary time, with no context or reminiscence of the sooner turns.
The API doesn’t retailer something between calls, so every request goes in chilly. The messages array is the way you repair that: pack the total dialog historical past into each request, person turns and assistant turns each, and the mannequin responds as if it remembers the entire thread. Go away it out and also you get that very same damaged expertise I shipped by chance.
const periods = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
perform getOrCreateSession(sessionId) {
if (!periods.has(sessionId)) {
periods.set(sessionId, {
messages: [],
lastActive: Date.now(),
});
}
const session = periods.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of periods) {
if (now - session.lastActive > SESSION_TTL_MS) {
periods.delete(id);
}
}
}, 10 * 60 * 1000);
The Map is ok at improvement scale and in single-process deployments. The lastActive timestamp will get bumped on each request. The cleanup interval runs each ten minutes and removes periods which have gone quiet for half an hour.
The exhausting restrict of this strategy is {that a} Node.js course of restart wipes it fully. In the event you run a number of cases behind a load balancer, every occasion has its personal Map, and a person whose requests hit totally different cases will lose their dialog mid-sentence. The repair is Redis: one shopper, similar information construction, TTL dealt with natively. I’m not going to construct that right here as a result of it will double the size of this text with out including something about streaming, however don’t ship this to manufacturing with out it.
The Streaming Endpoint
4 issues occur in sequence earlier than tokens begin flowing: enter validation, SSE setup, abort dealing with, and the OpenAI name with its token loop. Every step has not less than one non-obvious resolution.
SSE Setup
res.setHeader('Content material-Kind', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
flushHeaders() is the road I see lacking in additional streaming PRs than I can depend. Skip it and Categorical holds onto the response headers till the primary res.write() name. The browser opens the request, will get nothing again, sits there, and after just a few seconds the entire thing appears like a hung request. You’ll spend an hour checking your OpenAI name, your async logic, and your middleware, and the repair is one line on the high. Name flushHeaders() proper after setting the headers; nothing else modifications.
AbortController
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
One essential element: connect the shut listener to res, not req. The req object fires shut when the request physique is totally consumed, which occurs instantly after the POST physique arrives and would abort the stream earlier than it begins. res.on('shut') fires when the precise response connection drops, which is what you need. The !res.writableEnded guard prevents calling abort() after a clear end.
When the connection drops, controller.abort() propagates a cancellation sign to the OpenAI SDK and the for await loop exits on the following iteration. With out this, you retain pulling tokens from the API and writing them right into a closed socket, which implies you might be spending cash and reaching nothing.
The OpenAI Streaming Name
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{
role: 'system',
content: 'You are a helpful assistant. Be concise and clear.',
},
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
Set max_tokens. This isn’t elective. I left it uncapped throughout a testing session and watched my utilization dashboard spike in actual time. Left alone, the mannequin will write a dissertation in response to a three-word query. I exploit 800 for chat, which works high-quality for many issues, and you may tune it when you see what your customers truly ship.
The sign parameter threads the AbortController by way of to the SDK. If the shopper disconnects, the SDK cancels the underlying HTTP request to OpenAI, not simply the native loop.
Token Loop and Historical past
let fullResponse = '';
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`information: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
}
Every chunk accommodates one token in delta.content material. You concatenate them into fullResponse as they arrive, relay each to the shopper, and save the entire string to the session as soon as the loop ends.
The double newline in res.write() is necessary; it’s how SSE alerts the tip of a single occasion. The browser buffers information till it sees nn earlier than firing. Omit it and the shopper receives all tokens without delay when the connection closes, which is similar damaged conduct as not streaming in any respect.
Don’t save fullResponse if the stream was aborted. A truncated assistant reply within the session historical past will corrupt the dialog context, and the mannequin’s subsequent flip might be constructed on incomplete data.
The Full server.js
import 'dotenv/config';
import categorical from 'categorical';
import cors from 'cors';
import OpenAI from 'openai';
import path from 'path';
import crypto from 'crypto';
import { fileURLToPath } from 'url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const app = categorical();
const openai = new OpenAI({ apiKey: course of.env.OPENAI_API_KEY });
app.use(cors());
app.use(categorical.json());
app.use(categorical.static(path.be a part of(__dirname, 'public')));
const periods = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
perform getOrCreateSession(sessionId) {
if (!periods.has(sessionId)) {
periods.set(sessionId, { messages: [], lastActive: Date.now() });
}
const session = periods.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of periods) {
if (now - session.lastActive > SESSION_TTL_MS) periods.delete(id);
}
}, 10 * 60 * 1000);
app.submit('/session', (req, res) => {
const sessionId = crypto.randomUUID();
getOrCreateSession(sessionId);
console.log(`[/session] created: ${sessionId}`);
res.json({ sessionId });
});
app.delete('/session/:sessionId', (req, res) => {
periods.delete(req.params.sessionId);
res.json({ okay: true });
});
app.submit('/chat', async (req, res) => {
const { message, sessionId } = req.physique;
if (!message || typeof message !== 'string') {
return res.standing(400).json({ error: 'message is required' });
}
if (!sessionId || typeof sessionId !== 'string') {
return res.standing(400).json({ error: 'sessionId is required' });
}
const session = getOrCreateSession(sessionId);
session.messages.push({ function: 'person', content material: message });
console.log(`[/chat] obtained:`, { message, sessionId });
res.setHeader('Content material-Kind', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
let fullResponse = '';
attempt {
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{ role: 'system', content: 'You are a helpful assistant. Be concise and clear.' },
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`information: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
console.log(`[stream complete] ${fullResponse.size} chars`);
}
res.write(`information: ${JSON.stringify({ performed: true })}nn`);
res.finish();
} catch (err) {
if (err.title === 'AbortError' || err.title === 'APIUserAbortError' || err.message === 'Request was aborted.') {
res.finish();
return;
}
console.error('Stream error:', err.message);
res.write(`information: ${JSON.stringify({ error: err.message })}nn`);
res.finish();
}
});
const PORT = course of.env.PORT || 3000;
app.hear(PORT, () => console.log(`Server working at http://localhost:${PORT}`));
Constructing the Frontend
The shopper has two jobs: handle the session lifecycle and render the stream. The session lifecycle is straightforward (create on load, delete on “New Chat”), and the stream rendering is the place most implementations get tripped up.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta title="viewport" content material="width=device-width, initial-scale=1.0">
<title>AI Streaming Chat</title>
<model>
* { box-sizing: border-box; margin: 0; padding: 0; }
physique {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: #f0f2f5;
peak: 100vh;
show: flex;
flex-direction: column;
align-items: middle;
padding: 1.5rem 1rem;
}
.chat-wrapper { width: 100%; max-width: 700px; show: flex; flex-direction: column; peak: 100%; }
h1 { font-size: 1.2rem; coloration: #333; margin-bottom: 1rem; }
#messages {
flex: 1; overflow-y: auto; background: white; border: 1px stable #e0e0e0;
border-radius: 10px; padding: 1rem; show: flex; flex-direction: column; hole: 0.75rem;
}
.message {
max-width: 85%; padding: 0.65rem 0.9rem; border-radius: 10px;
line-height: 1.6; font-size: 0.93rem; white-space: pre-wrap; word-break: break-word;
}
.person { background: #0070f3; coloration: white; align-self: flex-end; }
.assistant { background: #f1f1f1; coloration: #111; align-self: flex-start; }
.input-row { show: flex; hole: 0.5rem; margin-top: 0.75rem; align-items: flex-end; }
textarea {
flex: 1; padding: 0.7rem; border: 1px stable #ddd; border-radius: 8px;
font-size: 0.93rem; font-family: inherit; resize: none; peak: 52px; overflow-y: auto;
}
button { padding: 0.65rem 1.2rem; border: none; border-radius: 8px; font-size: 0.93rem; cursor: pointer; }
#ship { background: #0070f3; coloration: white; }
#ship:disabled { background: #aaa; cursor: not-allowed; }
#clear { background: #f1f1f1; coloration: #555; }
.standing { font-size: 0.78rem; coloration: #aaa; margin-top: 0.3rem; min-height: 1rem; }
</model>
</head>
<physique>
<div class="chat-wrapper">
<h1>AI Chat</h1>
<div id="messages"></div>
<p class="standing" id="standing"></p>
<div class="input-row">
<textarea id="enter" placeholder="Ask one thing... (Ctrl+Enter to ship)"></textarea>
<button id="ship">Ship</button>
<button id="clear">New Chat</button>
</div>
</div>
<script>
let sessionId = null;
async perform initSession() {
const res = await fetch('/session', { methodology: 'POST' });
const information = await res.json();
sessionId = information.sessionId;
}
perform appendMessage(function, textual content) {
const div = doc.createElement('div');
div.className = `message ${function}`;
div.textContent = textual content;
doc.getElementById('messages').appendChild(div);
div.scrollIntoView({ conduct: 'clean' });
return div;
}
doc.getElementById('ship').addEventListener('click on', async () => {
const enter = doc.getElementById('enter');
const message = enter.worth.trim();
if (!message || !sessionId) return;
const sendBtn = doc.getElementById('ship');
const standing = doc.getElementById('standing');
enter.worth = '';
sendBtn.disabled = true;
standing.textContent = 'Producing...';
appendMessage('person', message);
const assistantBubble = appendMessage('assistant', '');
attempt {
const response = await fetch('/chat', {
methodology: 'POST',
headers: { 'Content material-Kind': 'software/json' },
physique: JSON.stringify({ message, sessionId }),
});
if (!response.okay) {
const err = await response.json();
assistantBubble.textContent = `Error: ${err.error}`;
return;
}
const reader = response.physique.getReader();
const decoder = new TextDecoder();
whereas (true) {
const { worth, performed } = await reader.learn();
if (performed) break;
const strains = decoder.decode(worth, { stream: true }).break up('n');
for (const line of strains) {
if (!line.startsWith('information: ')) proceed;
attempt {
const information = JSON.parse(line.slice(6));
if (information.performed) break;
if (information.error) { assistantBubble.textContent += `n[Error: ${data.error}]`; break; }
if (information.token) {
assistantBubble.textContent += information.token;
assistantBubble.scrollIntoView({ conduct: 'clean' });
}
} catch { }
}
}
standing.textContent = '';
} catch (err) {
assistantBubble.textContent = `Request failed: ${err.message}`;
} lastly {
sendBtn.disabled = false;
standing.textContent = '';
enter.focus();
}
});
doc.getElementById('clear').addEventListener('click on', async () => {
if (!sessionId) return;
await fetch(`/session/${sessionId}`, { methodology: 'DELETE' });
await initSession();
doc.getElementById('messages').innerHTML = '';
doc.getElementById('standing').textContent = '';
});
doc.getElementById('enter').addEventListener('keydown', (e) => {
if (e.key === 'Enter' && e.ctrlKey) doc.getElementById('ship').click on();
});
initSession();
</script>
</physique>
</html>
The attempt/catch contained in the inside loop handles an actual edge case: SSE occasions sometimes span two community reads. When that occurs, JSON.parse throws on the unfinished fragment, and catching it so the following learn can ship the remaining retains the stream alive. With out the catch, the entire stream crashes on what is actually regular community conduct.
Operating It
npm run dev
The terminal confirms the server began:
Server working at http://localhost:3000
Hit http://localhost:3000 in your browser and kind one thing. The terminal ought to print:
[/session] created: 2dfb2586-4601-4781-932d-cb3dcca5d5a5
[/chat] obtained: {
message: 'Are you aware about StackAbuse?',
sessionId: '2dfb2586-4601-4781-932d-cb3dcca5d5a5'
}
[stream complete] 304 chars
Within the browser, you will notice the response construct out token by token, which implies the streaming is working. The interface appears like this with a dwell response:
Ship a follow-up that references the primary reply and the mannequin will keep in mind it. The session historical past is included in each request, so context carries throughout turns. Click on “New Chat” and the server deletes the session and creates a contemporary one. The mannequin begins over with no reminiscence of the earlier dialog.
To examine the uncooked stream, open DevTools, go to Community, discover the /chat request, and take a look at the EventStream tab. You will note every information: occasion arrive individually, which is essentially the most helpful factor I do know of for debugging streaming conduct that appears right within the UI however just isn’t working the best way you suppose.
Manufacturing Gaps to Tackle Earlier than Launch
Earlier than you ship this, know what the implementation doesn’t deal with. I’ve seen builders take tutorial code and deploy it unchanged.
Context overflow. Each request sends the total session historical past. gpt-4o-mini has a 128,000-token context window, which is giant sufficient that almost all chat periods won’t ever hit it. However a session working for hours will ultimately strategy the restrict. When it does, the API returns a context_length_exceeded error. The usual repair is a sliding window: hold the system immediate, trim outdated messages from the entrance of the historical past, optionally run a summarization name to compress outdated context earlier than it falls off. This can be a actual engineering downside and there’s no one-size reply.
Fee limiting. There may be none right here, so one shopper can exhaust your month-to-month OpenAI quota. Add express-rate-limit earlier than this goes wherever close to the web.
Authentication. The /chat endpoint accepts requests from anybody. In an actual deployment, you want session-based auth or JWT validation earlier than the route handler runs. With out it, anybody who is aware of your URL is spending your API credit.
Multi-process periods. Lined above. Redis or equal earlier than you scale horizontally.
Conclusion
I’ve shipped variations of this precise setup to manufacturing twice now, throughout totally different corporations and merchandise, with the identical core sample: session retailer on the server, SSE for transport, and AbortController to cease losing cash on disconnected shoppers. The gaps I listed will not be hypothetical warnings; I hit most of them myself earlier than determining the fixes.
The half that journeys most builders just isn’t the OpenAI API itself, however the plumbing round it: getting tokens to the browser the second they arrive, protecting the connection alive with out burning credit on disconnected shoppers, and sustaining dialog context with out letting it develop unbounded. That’s the place the actual work sits. Hopefully this text saved you the afternoon I spent studying most of it the exhausting method.
The total code is at github.com/ziaongit/nodejs-openai-streaming. Clone it, break it, and make it your personal.
