Streams & Backpressure
The problem
Reading an entire large file or request body into memory before processing it wastes memory proportional to the data's size and delays any processing until the whole thing has arrived — streams let data be processed in small chunks as it arrives, but that only works safely if a fast producer can be told to slow down for a slow consumer.
Why now
File systems and I/O already established that reading and writing happens through the OS, often in chunks rather than all at once; streams are Node's abstraction over exactly that chunked reality, and libuv's event loop is what delivers each chunk asynchronously without blocking — backpressure is the missing piece neither of those alone provides: a signal that flows backward from consumer to producer.
Mental model
A stream is a sequence of chunks arriving over time instead of one big value arriving all at once — read one piece, do something with it, ask for the next. Backpressure is the stream's internal buffer telling the source 'stop sending, I'm full' the moment a consumer can't keep up, and 'resume' once there's room again — without it, a fast producer (a fast disk) writing to a slow consumer (a slow network client) would buffer unboundedly in memory until the process crashes.
Requires
- File Systems & I/OOperating Systems
Streams are Node's abstraction over exactly the chunked, asynchronous I/O file-systems-and-io already established happens at the OS level; you need that chunked reality as the underlying truth streams are modeling.
- libuv & the Event Loop PhasesNode.js Internals
Each stream chunk arrives as an event delivered through the poll phase already covered; you need the event loop's asynchronous delivery model to understand how and when stream 'data' events actually fire.
Used in
- Piping a large file directly to an HTTP response without loading it fully into memory
- Processing large CSV or log files chunk by chunk instead of reading them entirely upfront
- Video/audio streaming servers, where backpressure prevents unbounded memory growth
- Node's own HTTP request/response objects, which are themselves streams
Projects
- Stream a large file directly to an HTTP response using .pipe() and confirm via memory profiling that the whole file is never held in memory at once
- Deliberately write to a stream faster than its destination can drain, observe backpressure (write() returning false), and correctly pause/resume based on it
Examples
- fs.createReadStream(largeFile).pipe(response) streams a multi-gigabyte file to a client using a small, constant amount of memory, regardless of file size
- A slow network client causes a fast file-read stream's internal buffer to fill; without honoring backpressure, an unthrottled writer would keep buffering in memory until the process runs out
Resources
Mastery checklist
- I can pipe a readable stream to a writable stream and explain what's happening chunk by chunk
- I can explain what backpressure is and how a stream signals it
- I can implement correct pause/resume behavior when a write() call reports backpressure
- I can explain why streaming a large file uses constant memory while reading it fully does not