Streams

Process data piece-by-piece with Readable, Writable, and Transform streams to handle large files efficiently

Overview

Streams are one of Node.js's most powerful and distinguishing features. Rather than loading an entire file or HTTP response body into memory before processing it, streams let you work with data as it arrives — chunk by chunk. This makes it possible to process a 10 GB log file on a machine with only 512 MB of RAM, pipe a file from disk directly to an HTTP response without buffering it, or transform data on the fly as it passes through your application. Node.js has four stream types: Readable (a source of data — files, HTTP requests, stdin), Writable (a destination for data — files, HTTP responses, stdout), Transform (both readable and writable — gzip compression, encryption, JSON parsing), and Duplex (bidirectional, like TCP sockets). In practice you mostly use Readable and Writable, and occasionally Transform for data conversion pipelines. The key operation is piping: `readable.pipe(writable)` connects a source to a destination and handles backpressure automatically. Backpressure is the mechanism that slows the producer when the consumer can't keep up — without it, fast producers would overwhelm slow consumers by filling up memory. Piping is the cleanest way to connect streams and should be your default approach. Modern Node.js also provides the `pipeline` utility from the `stream` module (or its promise version from `stream/promises`), which improves on raw piping by correctly propagating errors and cleaning up all streams when any one fails. Using raw `.pipe()` can leak streams on error; `pipeline()` is the safer choice for production code.

Key Points

•Streams process data in chunks, allowing memory-efficient handling of arbitrarily large data
•Four types: Readable, Writable, Transform, Duplex
•readable.pipe(writable) connects streams and handles backpressure automatically
•stream/promises pipeline() is safer than .pipe() — it cleans up all streams on error
•Readable streams emit "data" events (flowing mode) or are read with .read() (paused mode); async iteration is the most ergonomic
•Transform streams implement _transform(chunk, encoding, callback) to convert data mid-pipe

Examples

Reading a large file with a Readable stream — memory efficient

const fs = require('fs');

// Reads file in ~64KB chunks, never loads it all at once
const readable = fs.createReadStream('large-file.log', { encoding: 'utf8' });

let lineCount = 0;
let partial   = '';

readable.on('data', (chunk) => {
  const text = partial + chunk;
  const lines = text.split('\n');
  partial = lines.pop(); // save incomplete last line
  lineCount += lines.length;
});

readable.on('end', () => {
  if (partial) lineCount++;
  console.log('Total lines:', lineCount);
});

readable.on('error', (err) => {
  console.error('Stream error:', err);
});

Even for a 10 GB file, memory usage stays low because only one chunk is in memory at a time.

Piping a file to an HTTP response — zero extra memory copies

const http = require('http');
const fs   = require('fs');
const path = require('path');

const server = http.createServer((req, res) => {
  const filePath = path.join(__dirname, 'public', req.url);

  // Pipe file directly to response — file is never loaded into memory
  const stream = fs.createReadStream(filePath);
  stream.on('error', () => {
    res.writeHead(404);
    res.end('Not Found');
  });

  res.writeHead(200);
  stream.pipe(res); // handles backpressure automatically
});

server.listen(3000);

Using pipeline() from stream/promises (recommended safe version)

const { pipeline } = require('stream/promises');
const fs   = require('fs');
const zlib = require('zlib');

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    zlib.createGzip(),       // Transform stream: compresses data
    fs.createWriteStream(output)
  );
  console.log(`Compressed ${input} → ${output}`);
}

// If any stream errors, pipeline cleans up all others automatically
compressFile('data.txt', 'data.txt.gz');

pipeline() propagates errors and destroys all streams in the chain on failure. Always prefer it over raw .pipe().

Async iteration over a readable stream — the most ergonomic modern approach

const fs = require('fs');

async function sumFile(filePath) {
  const stream = fs.createReadStream(filePath, { encoding: 'utf8' });
  let total = 0;

  for await (const chunk of stream) {
    // chunk is a string (because we set encoding)
    // In a real scenario: parse lines and sum values
    total += chunk.length;
  }

  return total;
}

sumFile('data.txt').then(t => console.log('Total chars:', t));

for await...of on a stream is equivalent to listening for "data" events but reads more like synchronous code.

Common Mistakes

Using raw .pipe() in production without error handlers — if the destination errors, the readable is not automatically destroyed, causing memory leaks
Collecting all stream chunks into an array and joining at the end — this defeats the purpose of streaming and is the same as readFile for memory usage
Not handling the "error" event on streams — uncaught stream errors crash the process with an unhandled error event
Mixing flowing mode (on("data")) and paused mode (stream.read()) on the same stream — pick one style

Note

Use `stream/promises pipeline` for any production stream pipeline. For development and one-off scripts, `for await...of` on a readable stream is the cleanest API. Use raw `on("data")` only when you need fine-grained control over chunk processing.

Streams

Overview

Key Points

Examples

Common Mistakes

Note

Exercises

Knowledge Check

What is backpressure in the context of Node.js streams?