Node Streams: Processing a 10 GB File on a 512 MB Server

Quick Answer

A stream processes data in chunks rather than loading it all at once, so memory use stays roughly constant regardless of file size. Use pipeline() rather than pipe() so errors propagate and resources are cleaned up.

The difference, measured

Counting the lines in a 75 MB CSV, two ways, each in its own process:

// whole.js
const data = fs.readFileSync("big.csv", "utf8");
const n = data.split("\n").length;

// streamed.js
const rl = readline.createInterface({ input: fs.createReadStream("big.csv") });
rl.on("line", () => n++);

readFileSync  lines: 3000000   peak RSS: 239 MB
streaming     lines: 3000000   peak RSS:  49 MB

Same answer, roughly five times the memory. The 239 MB is worse than the file size because the string, the split array and the buffer all coexist.

The important part is not the ratio but the shape. Streaming memory stays flat as the file grows; reading it whole grows with the file. At 10 GB the first approach does not get slower — it fails outright, because Node has a hard limit on string and buffer size well below that.

The four kinds of stream

• Readable — data comes out. A file being read, an incoming HTTP request.
• Writable — data goes in. A file being written, an HTTP response.
• Duplex — both, independently. A TCP socket.
• Transform — a duplex stream where output is a function of input. Compression, encryption, parsing.

You have already used them without noticing. process.stdout is writable. In Express, req is readable and res is writable — which is why you can pipe a file straight to a response:

app.get("/download", (req, res) => {
  fs.createReadStream("report.pdf").pipe(res);
});

That serves a file of any size using a small constant amount of memory, and it starts sending immediately rather than after reading everything.

Use pipeline, not pipe

pipe() is the familiar form and it has a real flaw: it does not forward errors. If the read fails midway, the writable stream is never closed and the file handle leaks. On a server, that accumulates until you run out of descriptors.

const { pipeline } = require("stream");
const { Transform } = require("stream");

const toUpper = new Transform({
  transform(chunk, enc, cb) {
    cb(null, chunk.toString().toUpperCase());
  },
});

pipeline(
  fs.createReadStream("in.txt"),
  toUpper,
  fs.createWriteStream("out.txt"),
  (err) => {
    if (err) console.error("failed:", err);
    else console.log("done");
  }
);

pipeline propagates errors to one callback and destroys every stream in the chain on failure. There is also a promise version in stream/promises for use with await.

The rule is simple: always use pipeline. pipe is acceptable only for a throwaway script where a leak does not matter.

Backpressure, which is the point

The reason streams work is not chunking alone — it is flow control.

Suppose you read from a fast local disk and write to a slow network. Without coordination, the reader produces faster than the writer consumes and the excess accumulates in memory. You are back to loading the whole file, just less obviously.

Streams handle this automatically. write() returns false when the internal buffer is full, and pipe and pipeline pause the source until the destination emits drain. Memory stays bounded because the fast end waits for the slow end.

This matters when writing streams by hand. Ignoring the return value of write() in a loop defeats backpressure entirely and reintroduces unbounded memory growth — a genuinely common bug in code that looks correct.

Where to use them

• Large file uploads and downloads. Stream to disk or to storage rather than buffering the whole body.
• CSV and log processing. Read, transform, write, in constant memory. Combine with readline for line-by-line work.
• Compression — zlib.createGzip() is a transform stream you drop into a pipeline.
• Proxying — pipe an upstream response straight to your client without buffering.
• Database exports — most drivers offer a cursor or stream mode instead of materialising every row.

Where not to bother: small files, JSON you must parse as a whole anyway, and anything where the complexity outweighs the benefit. Streaming a 4 KB config file adds nothing.

The signal is simple — if the data size is controlled by a user or grows over time, stream it. Anything sized by your own code and known to be small can be read whole.

Frequently Asked Questions