Stream Piping Basics: Connecting Data Flow in Node.js

Imagine you're moving water from one container to another. You could scoop it up bucket by bucket, carrying each load manually. Or you could connect a pipe between the containers and let the water flow automatically—no manual labor, no spills, just smooth continuous flow.

That's exactly what stream piping does in Node.js. It creates a direct connection that lets data flow seamlessly from one place to another, automatically handling all the details for you.

Today, we're going to discover one of Node.js's most elegant features. By the end of this journey, you'll understand how to move data efficiently and why piping is fundamental to building scalable applications.

Quick Reference

When to use: Connect readable and writable streams to transfer data automatically without manual buffering

Basic syntax:

readableStream.pipe(writableStream);

Common use cases:

Copying files efficiently
Streaming files to HTTP responses
Processing data in chunks
Handling large files without memory issues

Key benefit:

⚡ Processes 10GB file using only ~64KB memory
🚀 Starts processing immediately (no waiting)
🔄 Handles backpressure automatically

What You Need to Know First

To get the most out of this guide, you should understand:

Node.js streams basics: What readable and writable streams are and how they work (see our Introduction to Streams guide)
JavaScript async patterns: Understanding callbacks and basic event handling
File system operations: Basic familiarity with fs.readFileSync() and fs.writeFileSync()

If you're not comfortable with these topics, we recommend reviewing them first—especially the streams fundamentals, as piping builds directly on that knowledge.

What We'll Cover in This Article

By the end of this guide, you'll understand:

What stream piping is and why it exists
How to use .pipe() to connect streams
What happens behind the scenes when you pipe
How piping handles backpressure automatically
When to use piping vs loading data into memory
Basic piping patterns for common tasks

What We'll Explain Along the Way

Don't worry if you're unfamiliar with these—we'll explain them as we go:

Backpressure (how streams prevent memory overflow)
Stream events (pipe, unpipe, finish, end)
Chunks and buffers (how data flows through pipes)
Memory efficiency (why piping uses constant memory)

The Problem: Moving Large Amounts of Data

Before we learn about piping, let's understand the problem it solves. Imagine you need to copy a file. Here's the most obvious approach:

import fs from "fs";

// Read the entire file into memory
const data = fs.readFileSync("movie.mp4");

// Write all the data at once
fs.writeFileSync("movie-copy.mp4", data);

console.log("File copied!");

This works perfectly fine for small files. But what if movie.mp4 is 2GB? Let's see what happens:

The Problems:

Memory explosion: Your program needs 2GB of RAM just to hold the file
Long wait time: Nothing happens until the entire file is read
Server freeze: If this is a web server, it can't handle other requests while copying
Crash risk: If you don't have 2GB of free memory, your program crashes

Real-world impact:

// Copying a 2GB file
const data = fs.readFileSync("large-file.bin"); // Loads 2GB into memory!

// What happens:
// ❌ Memory usage spikes to 2GB+
// ❌ Process might crash: "JavaScript heap out of memory"
// ❌ Other requests are blocked
// ❌ Takes several seconds just to read the file

This is where streams and piping come to the rescue!

The Solution: Stream Piping

Piping solves the memory problem by processing data in small chunks instead of loading everything at once. Think of it like a conveyor belt—you don't need to hold all the boxes at once, just pass them along one at a time.

Here's the piping approach:

import fs from "fs";

// Create a readable stream (the source)
const readableStream = fs.createReadStream("movie.mp4");

// Create a writable stream (the destination)
const writableStream = fs.createWriteStream("movie-copy.mp4");

// Connect them with a pipe
readableStream.pipe(writableStream);

console.log("Copying started...");

What changes:

// With piping:
// ✅ Memory usage: ~64KB (regardless of file size!)
// ✅ Starts copying immediately
// ✅ Server remains responsive
// ✅ Handles 2GB files as easily as 2KB files

Let's see the difference visually:

// WITHOUT PIPING (loading into memory):
// [File: 2GB] → [Read ALL] → [RAM: 2GB] → [Write ALL] → [Copy: 2GB]
//              ↑ Wait...                   ↑ Wait...
//              Takes several seconds        Then several more

// WITH PIPING (streaming):
// [File: 2GB] → [Read 64KB] → [Write 64KB] → [Copy: 2GB]
//              ↓ Immediately  ↓ Immediately
//              [Read 64KB] → [Write 64KB]
//              ↓ Immediately  ↓ Immediately
//              [Read 64KB] → [Write 64KB]
//              ... continues until done
// Memory used at any moment: Only ~64KB!

See the magic? Instead of loading 2GB, we process it 64KB at a time. The data flows through like water through a pipe!

Your First Pipe: Step-by-Step

Let's build our first piped stream together. We'll copy a file from one location to another, and I'll explain every step.

Step 1: Understanding the Pieces

Before we connect anything, let's identify what we need:

import fs from "fs";

// Piece 1: The SOURCE (where data comes FROM)
const readableStream = fs.createReadStream("source.txt");
// Think of this as: "A tap that reads data from source.txt"
// It opens the file and prepares to read chunks

// Piece 2: The DESTINATION (where data goes TO)
const writableStream = fs.createWriteStream("destination.txt");
// Think of this as: "A drain that writes data to destination.txt"
// It opens/creates the file and prepares to write chunks

// Right now, we have two disconnected pieces:
// [source.txt] → 🚰 readableStream (not connected) writableStream 🚽 → [destination.txt]

What happens when you create these streams:

// When you create the readable stream:
const readable = fs.createReadStream("source.txt");
// 1. Node.js opens source.txt
// 2. It gets a "file descriptor" (like a handle to the file)
// 3. It's ready to read, but hasn't read anything yet
// 4. Memory used: Almost nothing (just the stream object)

// When you create the writable stream:
const writable = fs.createWriteStream("destination.txt");
// 1. Node.js creates/opens destination.txt
// 2. It gets a file descriptor for writing
// 3. It's ready to write, but hasn't written anything yet
// 4. Memory used: Almost nothing (just the stream object)

Step 2: Connecting the Pipe

Now comes the magic moment—we connect these two streams:

// The magic connection:
readableStream.pipe(writableStream);

// This single line does SO MUCH:
// 1. Tells readableStream: "When you get data, send it to writableStream"
// 2. Tells writableStream: "Expect data coming from readableStream"
// 3. Sets up automatic backpressure handling (more on this soon!)
// 4. Starts the data flow automatically

What happens behind the scenes:

readableStream.pipe(writableStream);

// Internally, Node.js does this (simplified):
readableStream.on("data", (chunk) => {
  // Got a chunk from the file!
  const canContinue = writableStream.write(chunk);

  if (!canContinue) {
    // Writable is overwhelmed, pause reading
    readableStream.pause();
  }
});

writableStream.on("drain", () => {
  // Writable is ready for more data
  readableStream.resume();
});

readableStream.on("end", () => {
  // No more data, close the writable
  writableStream.end();
});

// You don't have to write any of this—.pipe() does it all!

Step 3: Complete Working Example

Let's put it all together into a complete, working program:

import fs from "fs";

// Step 1: Create the source stream
console.log("📖 Opening source file...");
const readableStream = fs.createReadStream("input.txt");

// Step 2: Create the destination stream
console.log("📝 Creating destination file...");
const writableStream = fs.createWriteStream("output.txt");

// Step 3: Connect them
console.log("🔗 Connecting pipe...");
readableStream.pipe(writableStream);

// Step 4: Listen for completion
writableStream.on("finish", () => {
  console.log("✅ File copied successfully!");
});

// Optional: Track progress
let bytesRead = 0;
readableStream.on("data", (chunk) => {
  bytesRead += chunk.length;
  console.log(`📦 Copied ${bytesRead} bytes so far...`);
});

What you'll see when you run this:

📖 Opening source file...
📝 Creating destination file...
🔗 Connecting pipe...
📦 Copied 65536 bytes so far...
📦 Copied 131072 bytes so far...
📦 Copied 196608 bytes so far...
✅ File copied successfully!

Breaking down what happened:

Line 1: Opened input.txt for reading
Line 2: Created/opened output.txt for writing
Line 3: Connected them—data flow starts automatically!
During flow: Node.js reads 64KB chunks and writes them immediately
At end: The 'finish' event fires when all data is written

Understanding the Data Flow

Let's visualize exactly what happens when data flows through a pipe. This is crucial for understanding why piping is so powerful.

The Chunk-by-Chunk Journey

const readable = fs.createReadStream("data.txt");
const writable = fs.createWriteStream("copy.txt");

readable.pipe(writable);

// Let's follow ONE chunk through the pipe:

// TIME: 0ms
// readable: "I'm ready to read!"
// writable: "I'm ready to write!"

// TIME: 1ms
// readable reads from disk → Gets chunk: Buffer[64KB]
// readable: "I got data! Sending to writable..."

// TIME: 2ms
// writable receives → Buffer[64KB]
// writable: "Got it! Writing to disk..."

// TIME: 3ms
// writable writes to disk → Success!
// writable: "Ready for more!"

// TIME: 4ms
// readable reads next chunk → Buffer[64KB]
// ... the cycle continues ...

// This happens automatically until the file is completely copied!

Memory Usage Over Time

Let's compare memory usage between the two approaches:

// APPROACH 1: Load everything into memory
const data = fs.readFileSync("100MB-file.bin"); // 100MB
fs.writeFileSync("copy.bin", data);

// Memory usage over time:
// Time 0s:   10 MB (program starts)
// Time 1s:  110 MB (reading file)  ← Spike!
// Time 2s:  110 MB (still in memory)
// Time 3s:  110 MB (writing file)
// Time 4s:   10 MB (done, memory released)

// APPROACH 2: Piping
fs.createReadStream("100MB-file.bin").pipe(fs.createWriteStream("copy.bin"));

// Memory usage over time:
// Time 0s:   10 MB (program starts)
// Time 1s:   10 MB (streaming chunk 1)
// Time 2s:   10 MB (streaming chunk 2)
// Time 3s:   10 MB (streaming chunk 3)
// Time 4s:   10 MB (done)
// ↑ Stays constant! Memory usage doesn't change!

The key insight: With piping, memory usage stays constant regardless of file size!

Pipe Events: What's Happening Under the Hood

When you call .pipe(), several events start firing. Understanding these events helps you know exactly what's happening at each stage.

The Main Events

Let's watch all the events that fire during a pipe operation:

import fs from "fs";

const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("output.txt");

console.log("Setting up event listeners...\n");

// Readable stream events
readable.on("open", (fd) => {
  console.log("1️⃣ Readable: File opened (descriptor:", fd + ")");
});

readable.on("data", (chunk) => {
  console.log(`2️⃣ Readable: Read ${chunk.length} bytes`);
});

readable.on("end", () => {
  console.log("3️⃣ Readable: Finished reading all data");
});

readable.on("close", () => {
  console.log("4️⃣ Readable: Stream closed");
});

// Writable stream events
writable.on("pipe", (src) => {
  console.log("5️⃣ Writable: Pipe connected from", src.constructor.name);
});

writable.on("finish", () => {
  console.log("6️⃣ Writable: Finished writing all data");
});

writable.on("close", () => {
  console.log("7️⃣ Writable: Stream closed");
});

// Start the pipe!
console.log("Starting pipe...\n");
readable.pipe(writable);

Typical output:

Setting up event listeners...

Starting pipe...

1️⃣ Readable: File opened (descriptor: 20)
5️⃣ Writable: Pipe connected from ReadStream
2️⃣ Readable: Read 65536 bytes
2️⃣ Readable: Read 65536 bytes
2️⃣ Readable: Read 32768 bytes
3️⃣ Readable: Finished reading all data
6️⃣ Writable: Finished writing all data
7️⃣ Writable: Stream closed
4️⃣ Readable: Stream closed

Key Events Explained

Let's understand what each important event means and when you'd use it:

pipe event (on writable stream):

writable.on("pipe", (sourceStream) => {
  console.log("Someone just connected to me!");
  console.log("The source is:", sourceStream.constructor.name);
});

// Fires: Immediately when .pipe() is called
// Use for: Logging connections, tracking active pipes

data event (on readable stream):

readable.on("data", (chunk) => {
  console.log(`Received chunk: ${chunk.length} bytes`);
  // This is the actual data flowing through!
});

// Fires: Every time a chunk is read from the source
// Use for: Progress tracking, monitoring throughput
// Note: You usually don't need this—.pipe() handles it automatically

finish event (on writable stream):

writable.on("finish", () => {
  console.log("All data has been written!");
  // This is THE event to listen for completion
});

// Fires: After all data is written and stream.end() is called
// Use for: Knowing when the pipe is complete
// This is the most important event for detecting completion!

end event (on readable stream):

readable.on("end", () => {
  console.log("No more data to read");
  // The source has been exhausted
});

// Fires: When there's no more data to read
// Use for: Cleanup, logging, chaining operations

Practical Example: Progress Tracking

Here's a real-world use of events—showing copy progress:

import fs from "fs";

function copyFileWithProgress(source: string, dest: string): Promise<void> {
  return new Promise((resolve, reject) => {
    const readable = fs.createReadStream(source);
    const writable = fs.createWriteStream(dest);

    // Get file size for percentage calculation
    let totalBytes = 0;
    let copiedBytes = 0;

    fs.stat(source, (err, stats) => {
      if (err) {
        reject(err);
        return;
      }
      totalBytes = stats.size;
      console.log(`📁 File size: ${totalBytes} bytes\n`);
    });

    // Track progress
    readable.on("data", (chunk) => {
      copiedBytes += chunk.length;
      const percentage = ((copiedBytes / totalBytes) * 100).toFixed(1);
      process.stdout.write(`\r📊 Progress: ${percentage}%`);
    });

    // Detect completion
    writable.on("finish", () => {
      console.log("\n✅ Copy complete!");
      resolve();
    });

    // Handle errors (we'll cover this more in the next article)
    readable.on("error", reject);
    writable.on("error", reject);

    // Start the pipe
    readable.pipe(writable);
  });
}

// Usage:
copyFileWithProgress("large-file.zip", "copy.zip")
  .then(() => console.log("Done!"))
  .catch((err) => console.error("Failed:", err));

// Output:
// 📁 File size: 52428800 bytes
//
// 📊 Progress: 12.5%
// 📊 Progress: 25.0%
// 📊 Progress: 37.5%
// 📊 Progress: 50.0%
// 📊 Progress: 62.5%
// 📊 Progress: 75.0%
// 📊 Progress: 87.5%
// 📊 Progress: 100.0%
// ✅ Copy complete!
// Done!

Backpressure: The Automatic Traffic Control

One of the most powerful features of piping is automatic backpressure handling. Let's discover what this means and why it matters.

What Is Backpressure?

Imagine you're filling a bucket from a fire hose. If water comes too fast, the bucket overflows. You need to slow down the water flow to match the bucket's capacity.

The same thing happens with streams—sometimes the readable stream can produce data faster than the writable stream can consume it. This is called backpressure.

Without automatic handling:

// ❌ Manual approach (dangerous!)
readable.on("data", (chunk) => {
  writable.write(chunk); // What if writable can't keep up?
});

// Problem:
// - Readable keeps sending data
// - Writable's buffer fills up
// - Memory usage grows and grows
// - Eventually: Out of memory crash!

With piping (automatic handling):

// ✅ Piping approach (safe!)
readable.pipe(writable);

// Automatically handles:
// 1. When writable's buffer is full, pause readable
// 2. When writable's buffer drains, resume readable
// 3. Memory stays constant!

Seeing Backpressure in Action

Let's create a scenario where backpressure occurs and watch .pipe() handle it automatically:

import fs from "fs";

const readable = fs.createReadStream("large-file.bin", {
  highWaterMark: 1024 * 1024, // Read 1MB chunks (fast!)
});

const writable = fs.createWriteStream("output.bin", {
  highWaterMark: 16 * 1024, // Write buffer: 16KB (slow!)
});

// Monitor backpressure events
readable.on("pause", () => {
  console.log("⏸️  Readable PAUSED - writable can't keep up!");
});

readable.on("resume", () => {
  console.log("▶️  Readable RESUMED - writable caught up!");
});

// Monitor writable buffer state
let writeCount = 0;
const originalWrite = writable.write.bind(writable);
writable.write = function (chunk, ...args) {
  writeCount++;
  const canContinue = originalWrite(chunk, ...args);

  if (!canContinue) {
    console.log(`🛑 Write ${writeCount}: Buffer full! Backpressure applied.`);
  }

  return canContinue;
};

writable.on("drain", () => {
  console.log("💧 Writable buffer drained - ready for more!");
});

// Start the pipe
console.log("Starting pipe with backpressure monitoring...\n");
readable.pipe(writable);

writable.on("finish", () => {
  console.log("\n✅ Transfer complete!");
});

Output:

Starting pipe with backpressure monitoring...

🛑 Write 1: Buffer full! Backpressure applied.
⏸️  Readable PAUSED - writable can't keep up!
💧 Writable buffer drained - ready for more!
▶️  Readable RESUMED - writable caught up!
🛑 Write 2: Buffer full! Backpressure applied.
⏸️  Readable PAUSED - writable can't keep up!
💧 Writable buffer drained - ready for more!
▶️  Readable RESUMED - writable caught up!
...
✅ Transfer complete!

What's happening:

Readable reads 1MB chunks (very fast)
Writable can only buffer 16KB (much slower)
Writable's buffer fills up quickly
.pipe() automatically pauses the readable
Writable drains its buffer to disk
.pipe() automatically resumes the readable
Process repeats until done

The magic: You didn't have to write any of this logic—.pipe() does it all automatically!

Why Backpressure Matters

Let's see what happens without proper backpressure handling:

// ❌ WITHOUT backpressure handling:
const readable = fs.createReadStream("1GB-file.bin");
const writable = fs.createWriteStream("output.bin");

readable.on("data", (chunk) => {
  writable.write(chunk); // Ignoring the return value!
});

// What happens:
// Time 0s:  Memory: 50MB
// Time 1s:  Memory: 200MB  (chunks piling up in writable's buffer)
// Time 2s:  Memory: 400MB  (still piling up!)
// Time 3s:  Memory: 800MB  (danger!)
// Time 4s:  CRASH: "JavaScript heap out of memory"

// ✅ WITH backpressure handling (.pipe()):
readable.pipe(writable);

// What happens:
// Time 0s:  Memory: 50MB
// Time 1s:  Memory: 50MB  (constant)
// Time 2s:  Memory: 50MB  (constant)
// Time 3s:  Memory: 50MB  (constant)
// Time 4s:  Memory: 50MB  (completes successfully)

Key takeaway: Always use .pipe() instead of manually handling data events. It handles backpressure automatically and keeps your memory usage constant!

Common Piping Patterns

Let's explore real-world scenarios where stream piping shines. These are patterns you'll use again and again.

Pattern 1: File Copying

The most basic and common use case:

import fs from "fs";

// Simple copy
fs.createReadStream("source.txt").pipe(fs.createWriteStream("destination.txt"));

// With completion callback
function copyFile(source: string, dest: string): Promise<void> {
  return new Promise((resolve, reject) => {
    const readable = fs.createReadStream(source);
    const writable = fs.createWriteStream(dest);

    readable.pipe(writable);

    writable.on("finish", resolve);
    readable.on("error", reject);
    writable.on("error", reject);
  });
}

// Usage:
await copyFile("important.doc", "backup.doc");
console.log("✅ Backup created!");

Pattern 2: HTTP File Download

Streaming a file to an HTTP response (perfect for video streaming, large file downloads):

import express from "express";
import fs from "fs";

const app = express();

app.get("/download/:filename", (req, res) => {
  const filename = req.params.filename;
  const filepath = `./files/${filename}`;

  // Check if file exists
  if (!fs.existsSync(filepath)) {
    return res.status(404).send("File not found");
  }

  // Set headers
  res.setHeader("Content-Disposition", `attachment; filename="${filename}"`);

  // Pipe file to HTTP response
  // The HTTP response object is a writable stream!
  fs.createReadStream(filepath).pipe(res);

  console.log(`📤 Streaming ${filename} to client...`);
});

app.listen(3000, () => {
  console.log("Server running on http://localhost:3000");
});

// Why this is powerful:
// - 10GB file? No problem—uses only 64KB memory
// - Download starts immediately (no waiting to read entire file)
// - If client disconnects, stream stops automatically
// - Server can handle multiple downloads simultaneously

Pattern 3: Reading and Processing Data

Sometimes you need to see the data as it flows through:

import fs from "fs";

const readable = fs.createReadStream("data.txt");
const writable = fs.createWriteStream("processed.txt");

// Process data as it flows
let totalBytes = 0;
let lineCount = 0;

readable.on("data", (chunk) => {
  totalBytes += chunk.length;

  // Count lines in this chunk
  const lines = chunk.toString().split("\n").length - 1;
  lineCount += lines;

  console.log(`Processed ${totalBytes} bytes, ${lineCount} lines so far...`);
});

// Still pipe it through
readable.pipe(writable);

writable.on("finish", () => {
  console.log(`\n✅ Complete!`);
  console.log(`Total: ${totalBytes} bytes, ${lineCount} lines`);
});

Pattern 4: Conditional Piping

Sometimes you want to choose the destination dynamically:

import fs from "fs";

function saveToLocation(data: string, isBackup: boolean) {
  const readable = fs.createReadStream(data);

  // Choose destination based on condition
  const destination = isBackup
    ? fs.createWriteStream("backup/data.txt")
    : fs.createWriteStream("primary/data.txt");

  readable.pipe(destination);

  destination.on("finish", () => {
    console.log(`✅ Saved to ${isBackup ? "backup" : "primary"} location`);
  });
}

// Usage:
saveToLocation("temp-data.txt", true); // Saves to backup/
saveToLocation("temp-data.txt", false); // Saves to primary/

Common Misconceptions

Let's address some false beliefs that trip up developers when learning about piping.

❌ Misconception: "Piping loads all data into memory first"

Reality: Piping is specifically designed to avoid loading all data into memory. Data flows in small chunks.

Why this matters: This is the entire point of streams! If piping loaded everything into memory, streams would be pointless.

Proof:

import fs from "fs";

console.log(
  "Memory before:",
  process.memoryUsage().heapUsed / 1024 / 1024,
  "MB"
);

// Pipe a 5GB file
fs.createReadStream("5GB-file.bin").pipe(fs.createWriteStream("copy.bin"));

// Check memory every second
setInterval(() => {
  const memoryMB = process.memoryUsage().heapUsed / 1024 / 1024;
  console.log("Memory during copy:", memoryMB.toFixed(2), "MB");
}, 1000);

// Output:
// Memory before: 15.23 MB
// Memory during copy: 15.45 MB
// Memory during copy: 15.67 MB
// Memory during copy: 15.43 MB
// ... (stays around 15-16 MB regardless of file size!)

❌ Misconception: "Piping is only for files"

Reality: Piping works with ANY readable/writable stream pair: HTTP, sockets, compression, encryption, and more.

Examples:

// HTTP request to file
http.get("http://example.com/data.json", (response) => {
  response.pipe(fs.createWriteStream("downloaded.json"));
});

// File to HTTP response
app.get("/video", (req, res) => {
  fs.createReadStream("movie.mp4").pipe(res);
});

// Stdin to stdout (echo program)
process.stdin.pipe(process.stdout);

// File to compression to file
import zlib from "zlib";
fs.createReadStream("file.txt")
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream("file.txt.gz"));

❌ Misconception: "I can't see the data while piping"

Reality: You can listen to the data event to inspect data as it flows, without interrupting the pipe.

Example:

const readable = fs.createReadStream("data.txt");
const writable = fs.createWriteStream("copy.txt");

// Inspect data without interrupting flow
readable.on("data", (chunk) => {
  console.log("Saw", chunk.length, "bytes pass through");
  // Data still flows to writable!
});

readable.pipe(writable);

❌ Misconception: "Piping makes code asynchronous"

Reality: Streams are already asynchronous. Piping just connects them—it doesn't change the async nature.

Understanding:

const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("output.txt");

readable.pipe(writable);

console.log("This prints immediately!");
// Piping starts the process but doesn't block
// The copying happens in the background

writable.on("finish", () => {
  console.log("This prints when copying is done");
});

When to Use Piping

Now that you understand how piping works, when should you use it?

✅ Use Piping When:

1. Working with large files (> 10MB):

// ✅ Good: 1GB video file
fs.createReadStream("movie.mp4").pipe(fs.createWriteStream("copy.mp4"));
// Memory: ~64KB

// ❌ Bad: 1GB video file
const data = fs.readFileSync("movie.mp4");
fs.writeFileSync("copy.mp4", data);
// Memory: 1GB+ (might crash!)

2. Streaming data to/from HTTP:

// ✅ Download with streams
http.get("http://example.com/largefile.zip", (response) => {
  response.pipe(fs.createWriteStream("download.zip"));
});

// ✅ Upload with streams
app.get("/video", (req, res) => {
  fs.createReadStream("video.mp4").pipe(res);
});

3. Processing data in real-time:

// ✅ Process log file as it's being written
const logStream = fs.createReadStream("app.log");
logStream.on("data", (chunk) => {
  analyzeLogData(chunk); // Process immediately
});
logStream.pipe(fs.createWriteStream("processed.log"));

4. Memory-constrained environments:

// ✅ Raspberry Pi with 512MB RAM handling 2GB file
// No problem with piping—uses only 64KB!
fs.createReadStream("huge-data.bin").pipe(fs.createWriteStream("output.bin"));

❌ Don't Bother with Piping When:

1. Files are very small (< 1MB):

// Small config file—simpler to just read it
const config = JSON.parse(fs.readFileSync("config.json", "utf8"));

// Piping would work but adds unnecessary complexity:
let configData = "";
fs.createReadStream("config.json")
  .on("data", (chunk) => {
    configData += chunk;
  })
  .on("end", () => {
    const config = JSON.parse(configData);
  });
// ^ Too much code for a small file!

2. You need the complete data before processing:

// ❌ Can't use piping: Need to parse entire JSON first
const data = JSON.parse(fs.readFileSync("data.json"));
const result = data.users.map((u) => u.name); // Need full object

// With streams, you'd need to buffer everything anyway:
let buffer = "";
readable.on("data", (chunk) => {
  buffer += chunk;
});
readable.on("end", () => {
  const data = JSON.parse(buffer); // Defeats the purpose!
});

3. Simple one-time scripts:

// Quick script to copy one file once
// Readability > optimization
fs.copyFileSync("a.txt", "b.txt"); // Simple and clear

// Streaming version is overkill:
fs.createReadStream("a.txt")
  .pipe(fs.createWriteStream("b.txt"))
  .on("finish", () => console.log("Done"));

Practical Examples

Let's build some real-world utilities using piping.

Example 1: File Copy Utility with Progress

import fs from "fs";
import path from "path";

/**
 * Copies a file with progress tracking
 */
function copyFileWithProgress(
  sourcePath: string,
  destPath: string
): Promise<void> {
  return new Promise((resolve, reject) => {
    // Validate source exists
    if (!fs.existsSync(sourcePath)) {
      return reject(new Error(`Source file not found: ${sourcePath}`));
    }

    // Get file info
    const stats = fs.statSync(sourcePath);
    const totalBytes = stats.size;
    let copiedBytes = 0;

    console.log(`\n📁 Copying: ${path.basename(sourcePath)}`);
    console.log(`📊 Size: ${(totalBytes / 1024 / 1024).toFixed(2)} MB\n`);

    // Create streams
    const readable = fs.createReadStream(sourcePath);
    const writable = fs.createWriteStream(destPath);

    // Track progress
    const startTime = Date.now();
    readable.on("data", (chunk) => {
      copiedBytes += chunk.length;
      const percentage = ((copiedBytes / totalBytes) * 100).toFixed(1);
      const elapsed = (Date.now() - startTime) / 1000;
      const speed = (copiedBytes / elapsed / 1024 / 1024).toFixed(2);

      process.stdout.write(
        `\r⏳ Progress: ${percentage}% | Speed: ${speed} MB/s`
      );
    });

    // Handle completion
    writable.on("finish", () => {
      const duration = ((Date.now() - startTime) / 1000).toFixed(2);
      console.log(`\n✅ Copy complete in ${duration}s`);
      resolve();
    });

    // Handle errors
    readable.on("error", (err) => {
      console.error("\n❌ Read error:", err.message);
      writable.destroy();
      reject(err);
    });

    writable.on("error", (err) => {
      console.error("\n❌ Write error:", err.message);
      readable.destroy();
      reject(err);
    });

    // Start the copy
    readable.pipe(writable);
  });
}

// Usage:
async function main() {
  try {
    await copyFileWithProgress("large-video.mp4", "backup-video.mp4");
    console.log("🎉 Backup created successfully!");
  } catch (err) {
    console.error("Failed to create backup:", err.message);
    process.exit(1);
  }
}

main();

// Output:
// 📁 Copying: large-video.mp4
// 📊 Size: 524.28 MB
//
// ⏳ Progress: 100.0% | Speed: 87.38 MB/s
// ✅ Copy complete in 6.00s
// 🎉 Backup created successfully!

Example 2: Simple File Server

import http from "http";
import fs from "fs";
import path from "path";

const PORT = 3000;
const FILES_DIR = "./public";

const server = http.createServer((req, res) => {
  // Only handle GET requests
  if (req.method !== "GET") {
    res.writeHead(405, { "Content-Type": "text/plain" });
    res.end("Method Not Allowed");
    return;
  }

  // Build file path (remove leading slash)
  const filename = req.url === "/" ? "index.html" : req.url.slice(1);
  const filepath = path.join(FILES_DIR, filename);

  // Security: Prevent directory traversal
  if (!filepath.startsWith(path.resolve(FILES_DIR))) {
    res.writeHead(403, { "Content-Type": "text/plain" });
    res.end("Forbidden");
    return;
  }

  // Check if file exists
  fs.stat(filepath, (err, stats) => {
    if (err) {
      res.writeHead(404, { "Content-Type": "text/plain" });
      res.end("File Not Found");
      console.log(`❌ 404: ${filename}`);
      return;
    }

    // Set headers
    res.writeHead(200, {
      "Content-Type": getContentType(filename),
      "Content-Length": stats.size,
    });

    // Stream the file to response
    const readable = fs.createReadStream(filepath);

    readable.pipe(res);

    console.log(
      `✅ Streaming: ${filename} (${(stats.size / 1024).toFixed(2)} KB)`
    );

    // Handle errors
    readable.on("error", (err) => {
      console.error(`❌ Stream error for ${filename}:`, err.message);
      res.end();
    });
  });
});

function getContentType(filename: string): string {
  const ext = path.extname(filename).toLowerCase();
  const types: Record<string, string> = {
    ".html": "text/html",
    ".css": "text/css",
    ".js": "application/javascript",
    ".json": "application/json",
    ".png": "image/png",
    ".jpg": "image/jpeg",
    ".gif": "image/gif",
    ".mp4": "video/mp4",
    ".pdf": "application/pdf",
  };
  return types[ext] || "application/octet-stream";
}

server.listen(PORT, () => {
  console.log(`🚀 File server running on http://localhost:${PORT}`);
  console.log(`📁 Serving files from: ${path.resolve(FILES_DIR)}`);
});

// Why piping is perfect here:
// - Can serve files of ANY size without memory issues
// - Multiple clients can download simultaneously
// - Efficient: Uses ~64KB per connection
// - If client disconnects, stream stops automatically

Example 3: Data Duplicator

import fs from "fs";

/**
 * Duplicates data to multiple destinations simultaneously
 */
class DataDuplicator {
  private readable: fs.ReadStream;
  private destinations: fs.WriteStream[] = [];

  constructor(sourcePath: string) {
    this.readable = fs.createReadStream(sourcePath);

    // Manually handle data to send to multiple destinations
    this.readable.on("data", (chunk) => {
      this.destinations.forEach((dest) => {
        dest.write(chunk);
      });
    });

    this.readable.on("end", () => {
      this.destinations.forEach((dest) => {
        dest.end();
      });
    });
  }

  addDestination(destPath: string): void {
    const writable = fs.createWriteStream(destPath);
    this.destinations.push(writable);
    console.log(`➕ Added destination: ${destPath}`);
  }

  start(): Promise<void> {
    return new Promise((resolve, reject) => {
      let completed = 0;
      const total = this.destinations.length;

      this.destinations.forEach((dest, index) => {
        dest.on("finish", () => {
          completed++;
          console.log(`✅ Destination ${index + 1}/${total} complete`);

          if (completed === total) {
            console.log("🎉 All destinations complete!");
            resolve();
          }
        });

        dest.on("error", reject);
      });

      this.readable.on("error", reject);
    });
  }
}

// Usage: Create 3 copies of a file simultaneously
async function main() {
  const duplicator = new DataDuplicator("important-data.txt");

  duplicator.addDestination("backup1/data.txt");
  duplicator.addDestination("backup2/data.txt");
  duplicator.addDestination("backup3/data.txt");

  await duplicator.start();
  console.log("All backups created!");
}

main().catch(console.error);

// Output:
// ➕ Added destination: backup1/data.txt
// ➕ Added destination: backup2/data.txt
// ➕ Added destination: backup3/data.txt
// ✅ Destination 1/3 complete
// ✅ Destination 2/3 complete
// ✅ Destination 3/3 complete
// 🎉 All destinations complete!
// All backups created!

Check Your Understanding

Let's test what you've learned!

Quick Quiz

1. What's the main benefit of piping compared to loading files into memory?

Show Answer

Memory efficiency and constant memory usage.

Piping processes data in small chunks (typically 64KB), so:

A 10GB file uses only ~64KB of memory
Memory usage stays constant regardless of file size
No risk of "out of memory" errors
Server can handle multiple operations simultaneously

Comparison:

// Loading into memory: Uses full file size in RAM
const data = fs.readFileSync("5GB.bin"); // Uses 5GB RAM!

// Piping: Uses constant small amount
fs.createReadStream("5GB.bin").pipe(fs.createWriteStream("copy.bin")); // Uses ~64KB RAM

2. What does this code do?

const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("output.txt");
readable.pipe(writable);
console.log("Done!");

Show Answer

It starts copying but the "Done!" message is incorrect.

The console.log('Done!') executes immediately, not when the copy is finished. Piping is asynchronous!

Correct version:

const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("output.txt");

readable.pipe(writable);

writable.on("finish", () => {
  console.log("Done!"); // ✅ Now this is accurate
});

// Or wrap in a Promise:
function copyFile(src, dest) {
  return new Promise((resolve, reject) => {
    const readable = fs.createReadStream(src);
    const writable = fs.createWriteStream(dest);

    readable.pipe(writable);
    writable.on("finish", resolve);
    readable.on("error", reject);
    writable.on("error", reject);
  });
}

await copyFile("input.txt", "output.txt");
console.log("Done!"); // ✅ Guaranteed to be done

3. True or False: Piping automatically handles backpressure

Show Answer

True!

This is one of the most important features of .pipe().

What backpressure means: When the destination (writable) can't keep up with the source (readable), .pipe() automatically:

Pauses the readable stream
Waits for the writable to catch up
Resumes the readable stream

Without piping:

// ❌ Manual approach - dangerous!
readable.on("data", (chunk) => {
  writable.write(chunk); // Ignoring return value = memory leak!
});

With piping:

// ✅ Automatic backpressure handling
readable.pipe(writable);
// Handles everything automatically!

You can see it in action:

readable.on("pause", () => console.log("Paused due to backpressure"));
readable.on("resume", () => console.log("Resumed"));
readable.pipe(writable);

4. When should you NOT use piping?

Show Answer

Don't use piping when:

Files are very small (< 1MB):

// Small file - simpler approach is fine
const data = fs.readFileSync("small-config.json");

You need the complete data before processing:

// Need entire JSON parsed first
const data = JSON.parse(fs.readFileSync("data.json"));
const names = data.users.map((u) => u.name);

One-time simple scripts:

// Quick script - readability matters more
fs.copyFileSync("a.txt", "b.txt");

Use piping when:

Files are large (> 10MB)
Memory is limited
Processing data in real-time
Building servers that handle multiple requests
Working with HTTP streams

Hands-On Exercise

Challenge: Create a simple file copy function that:

Copies a file using streams
Shows progress percentage
Returns a Promise
Handles errors properly

Starter Code:

import fs from "fs";

async function copyFile(source: string, dest: string): Promise<void> {
  // TODO: Implement this function
  // - Use fs.createReadStream and fs.createWriteStream
  // - Track progress with 'data' event
  // - Use 'finish' event to detect completion
  // - Handle errors from both streams
}

// Test it:
copyFile("test.txt", "test-copy.txt")
  .then(() => console.log("✅ Success!"))
  .catch((err) => console.error("❌ Failed:", err.message));

Solution:

Show Solution

import fs from "fs";

/**
 * Copies a file with progress tracking
 * @param source - Source file path
 * @param dest - Destination file path
 * @returns Promise that resolves when copy is complete
 */
async function copyFile(source: string, dest: string): Promise<void> {
  return new Promise((resolve, reject) => {
    // Step 1: Get file size for progress calculation
    let totalSize = 0;
    let copiedSize = 0;

    try {
      const stats = fs.statSync(source);
      totalSize = stats.size;
      console.log(`📁 File size: ${(totalSize / 1024).toFixed(2)} KB`);
    } catch (err) {
      return reject(new Error(`Cannot read source: ${err.message}`));
    }

    // Step 2: Create streams
    const readable = fs.createReadStream(source);
    const writable = fs.createWriteStream(dest);

    // Step 3: Track progress
    readable.on("data", (chunk) => {
      copiedSize += chunk.length;
      const percentage = ((copiedSize / totalSize) * 100).toFixed(1);
      process.stdout.write(`\r📊 Progress: ${percentage}%`);
    });

    // Step 4: Handle completion
    writable.on("finish", () => {
      console.log("\n✅ Copy complete!");
      resolve();
    });

    // Step 5: Handle errors
    readable.on("error", (err) => {
      console.error("\n❌ Read error:", err.message);
      writable.destroy(); // Clean up writable
      reject(err);
    });

    writable.on("error", (err) => {
      console.error("\n❌ Write error:", err.message);
      readable.destroy(); // Clean up readable
      reject(err);
    });

    // Step 6: Start the pipe
    readable.pipe(writable);
  });
}

// Create test file
fs.writeFileSync("test.txt", "Hello, World!\n".repeat(10000));

// Test the function
copyFile("test.txt", "test-copy.txt")
  .then(() => {
    console.log("✅ Test passed!");

    // Verify files are identical
    const source = fs.readFileSync("test.txt");
    const dest = fs.readFileSync("test-copy.txt");

    if (source.equals(dest)) {
      console.log("✅ Files are identical!");
    } else {
      console.log("❌ Files differ!");
    }
  })
  .catch((err) => {
    console.error("❌ Test failed:", err.message);
  });

Output:

📁 File size: 136.72 KB
📊 Progress: 100.0%
✅ Copy complete!
✅ Test passed!
✅ Files are identical!

Key points in the solution:

Get file size first - For accurate progress tracking
Create both streams - Source and destination
Track progress - Using 'data' event on readable
Detect completion - Using 'finish' event on writable
Handle ALL errors - Both readable and writable
Clean up on error - Destroy the other stream if one fails
Return Promise - Makes it easy to use with async/await

Summary: Key Takeaways

Let's review what we've discovered about stream piping:

What Piping Is:

✅ A method to automatically connect readable and writable streams
✅ Moves data in small chunks (typically 64KB) instead of loading everything
✅ Handles backpressure automatically to prevent memory overflow
✅ The most memory-efficient way to move data in Node.js

How to Use Piping:

// Basic syntax
readableStream.pipe(writableStream);

// With completion handling
readableStream.pipe(writableStream);
writableStream.on("finish", () => {
  console.log("Done!");
});

// With error handling (important!)
readableStream.on("error", handleError);
writableStream.on("error", handleError);

When to Use Piping:

✅ Large files (> 10MB)
✅ HTTP file uploads/downloads
✅ Memory-constrained environments
✅ Real-time data processing
✅ Multiple simultaneous operations

Key Benefits:

🚀 Constant memory usage - 10GB file uses only 64KB RAM
⚡ Immediate processing - Starts working instantly
🔄 Automatic backpressure - Prevents memory overflow
📦 Simple API - Just call .pipe()

Important Events:

// On readable stream
readable.on("data", (chunk) => {
  /* Track progress */
});
readable.on("end", () => {
  /* Reading complete */
});

// On writable stream
writable.on("pipe", (src) => {
  /* Pipe connected */
});
writable.on("finish", () => {
  /* Writing complete */
});

Remember:

Piping is asynchronous - always listen for 'finish' event
Backpressure is automatic - streams pause/resume as needed
Memory usage is constant - regardless of data size
Always handle errors - streams don't propagate errors automatically

Additional Resources

Official Documentation:

Node.js Stream API - Complete API reference
Stream Implementers Guide - Building custom streams

Community Resources:

Stream Handbook - Comprehensive guide by James Halliday
Node.js Streams: Everything you need to know - FreeCodeCamp article

Useful Libraries:

pump - Pipe with proper error handling
through2 - Simplified transform stream creation
split2 - Split streams by newlines

Version Information

Tested with:

Node.js: v18.x, v20.x, v22.x
Works in: Node.js environment (streams are not available in standard browser environments)

API Compatibility:

.pipe(): Available since Node.js v0.10.0
Basic streams API: Stable since Node.js v10.0.0

Known Issues:

⚠️ .pipe() does not automatically propagate errors (we'll cover pipeline() in the next article)
⚠️ Multiple pipes to the same writable need careful handling

What's Coming:

🔮 Next article will cover pipeline() for better error handling
🔮 We'll explore chaining multiple streams together
🔮 You'll learn to build custom transform streams

Quick Reference​

What You Need to Know First​

What We'll Cover in This Article​

What We'll Explain Along the Way​

The Problem: Moving Large Amounts of Data​

The Solution: Stream Piping​

Your First Pipe: Step-by-Step​

Step 1: Understanding the Pieces​

Step 2: Connecting the Pipe​

Step 3: Complete Working Example​

Understanding the Data Flow​

The Chunk-by-Chunk Journey​

Memory Usage Over Time​

Pipe Events: What's Happening Under the Hood​

The Main Events​

Key Events Explained​

Practical Example: Progress Tracking​

Backpressure: The Automatic Traffic Control​

What Is Backpressure?​

Seeing Backpressure in Action​

Why Backpressure Matters​

Common Piping Patterns​

Pattern 1: File Copying​

Pattern 2: HTTP File Download​

Pattern 3: Reading and Processing Data​

Pattern 4: Conditional Piping​

Common Misconceptions​

❌ Misconception: "Piping loads all data into memory first"​

❌ Misconception: "Piping is only for files"​

❌ Misconception: "I can't see the data while piping"​

❌ Misconception: "Piping makes code asynchronous"​

When to Use Piping​

✅ Use Piping When:​

❌ Don't Bother with Piping When:​

Practical Examples​

Example 1: File Copy Utility with Progress​

Example 2: Simple File Server​

Example 3: Data Duplicator​

Check Your Understanding​

Quick Quiz​

Hands-On Exercise​

Summary: Key Takeaways​

Additional Resources​

Version Information​

Quick Reference

What You Need to Know First

What We'll Cover in This Article

What We'll Explain Along the Way

The Problem: Moving Large Amounts of Data

The Solution: Stream Piping

Your First Pipe: Step-by-Step

Step 1: Understanding the Pieces

Step 2: Connecting the Pipe

Step 3: Complete Working Example

Understanding the Data Flow

The Chunk-by-Chunk Journey

Memory Usage Over Time

Pipe Events: What's Happening Under the Hood

The Main Events

Key Events Explained

Practical Example: Progress Tracking

Backpressure: The Automatic Traffic Control

What Is Backpressure?

Seeing Backpressure in Action

Why Backpressure Matters

Common Piping Patterns

Pattern 1: File Copying

Pattern 2: HTTP File Download

Pattern 3: Reading and Processing Data

Pattern 4: Conditional Piping

Common Misconceptions

❌ Misconception: "Piping loads all data into memory first"

❌ Misconception: "Piping is only for files"

❌ Misconception: "I can't see the data while piping"

❌ Misconception: "Piping makes code asynchronous"

When to Use Piping

✅ Use Piping When:

❌ Don't Bother with Piping When:

Practical Examples

Example 1: File Copy Utility with Progress

Example 2: Simple File Server

Example 3: Data Duplicator

Check Your Understanding

Quick Quiz

Hands-On Exercise

Summary: Key Takeaways

Additional Resources

Version Information