Project: CSV Reader with Streams

Build a streaming CSV parser using Node.js Readable streams, Transform streams, and the fs module

Overview

This project teaches you to think in streams by building a CSV parser entirely from Node.js built-in modules — no npm packages allowed. Real CSV files can be hundreds of megabytes and cannot be fully loaded into memory on constrained systems. Streams are how professional Node.js tools (databases, ETL pipelines, log processors) handle this problem, and CSV is a perfect vehicle for learning the pattern because the file format is just text split by newlines and commas. You will build a pipeline that reads a large CSV file chunk by chunk, splits the raw binary data into lines across chunk boundaries, parses each line into an object keyed by the header row, and emits the parsed objects as events or collects them into an array. You will compose three stream types: a Readable (fs.createReadStream), a Transform (your line splitter), and a Writable (your result consumer). The trickiest part of stream-based parsing is chunk boundaries — a single line can be split across two chunks, so you cannot process each chunk in isolation. You will implement a leftover buffer pattern: keep the incomplete partial line from the current chunk, prepend it to the next chunk, and only process complete lines. This pattern appears in countless real Node.js parsers and is worth understanding deeply. By the end of this project you will have a reusable CSV parser that handles arbitrarily large files, supports quoted fields with embedded commas, and exposes both a streaming interface (row events) and a promise-based interface (parseFile → returns array of objects).

Key Points

•fs.createReadStream reads a file in configurable chunks (default 64 KB) rather than loading it all into memory
•Transform streams sit in the middle of a pipeline — they receive Buffer chunks, transform them, and push results downstream
•The leftover/remainder buffer pattern handles lines split across chunk boundaries: keep incomplete data, prepend to the next chunk
•CSV fields can contain commas if they are wrapped in double quotes ("New York, NY") — a naive split(",") breaks on these
•pipe() connects a Readable to a Writable; Transform implements both and can be chained in the middle
•Emit row events from your Transform stream so consumers can process data as it arrives, before the file finishes reading

Examples

Final CSV reader — both streaming and batch interfaces

// Sample file: users.csv
// id,name,city
// 1,Alice,"New York, NY"
// 2,Bob,London
// 3,Charlie,"Paris, France"

const { parseFile, createCsvStream } = require('./csv-reader');

// Batch API — collect all rows into an array
const rows = await parseFile('users.csv');
console.log(rows);
// [
//   { id: '1', name: 'Alice', city: 'New York, NY' },
//   { id: '2', name: 'Bob',   city: 'London' },
//   { id: '3', name: 'Charlie', city: 'Paris, France' },
// ]

// Streaming API — process rows as they arrive
const stream = createCsvStream('large-file.csv');
stream.on('row', row => console.log(row));
stream.on('end', () => console.log('Done'));
stream.on('error', err => console.error(err));

Naive CSV split (breaks on quoted commas) vs. correct approach

// ❌ Breaks on: 1,Alice,"New York, NY"
const fields = line.split(',');
// → ['1', 'Alice', '"New York', ' NY"']  ← wrong

// ✅ Correct: parse character by character
function parseCsvLine(line) {
  const fields = [];
  let current  = '';
  let inQuotes = false;

  for (const char of line) {
    if (char === '"') {
      inQuotes = !inQuotes;
    } else if (char === ',' && !inQuotes) {
      fields.push(current.trim());
      current = '';
    } else {
      current += char;
    }
  }
  fields.push(current.trim()); // last field
  return fields;
}

Transform stream with the leftover buffer pattern

const { Transform } = require('stream');

class CsvTransform extends Transform {
  constructor(options = {}) {
    super({ ...options, objectMode: true }); // push objects, not buffers
    this._remainder = '';
    this._headers   = null;
  }

  _transform(chunk, _encoding, callback) {
    const data  = this._remainder + chunk.toString();
    const lines = data.split('\n');

    // Keep the last (potentially incomplete) line for next chunk
    this._remainder = lines.pop();

    for (const line of lines) {
      if (!line.trim()) continue; // skip blank lines
      const fields = parseCsvLine(line);

      if (!this._headers) {
        this._headers = fields;   // first row = header
      } else {
        const row = {};
        this._headers.forEach((h, i) => { row[h] = fields[i] ?? ''; });
        this.push(row); // push object downstream
      }
    }
    callback();
  }

  _flush(callback) {
    // Process last line (no trailing newline in many files)
    if (this._remainder.trim() && this._headers) {
      const fields = parseCsvLine(this._remainder);
      const row    = {};
      this._headers.forEach((h, i) => { row[h] = fields[i] ?? ''; });
      this.push(row);
    }
    callback();
  }
}

Batch API using stream/promises pipeline

const { pipeline } = require('stream/promises');
const fs = require('fs');

async function parseFile(filePath) {
  const rows      = [];
  const readable  = fs.createReadStream(filePath);
  const transform = new CsvTransform();

  transform.on('data', row => rows.push(row));
  await pipeline(readable, transform);
  return rows;
}

Common Mistakes

Calling line.split(",") on CSV lines — this breaks whenever a field contains a comma inside quotes; always use a proper char-by-char parser
Forgetting to implement _flush — the very last line of a CSV file often has no trailing newline, so the remainder buffer is never flushed without an explicit _flush method
Not setting objectMode: true in the Transform constructor — without it, Node.js expects Buffer or string outputs, and pushing a plain object throws a TypeError
Calling callback() before finishing all processing in _transform — the callback signals "I am ready for more data"; calling it early causes data to arrive before you have processed the current chunk

Note

Pass `{ highWaterMark: 1024 * 1024 }` to fs.createReadStream to read 1 MB chunks instead of the default 64 KB. For large files this reduces the number of _transform calls and can double parse speed.

Project: CSV Reader with Streams

Overview

Key Points

Examples

Common Mistakes

Note

Exercises

Knowledge Check

What is the purpose of the "remainder" buffer in a streaming line parser?