Buffer Operations: Reading, Writing, and Manipulating Binary Data

Once you've created a Buffer, you need to know how to work with it—reading bytes, writing data, copying sections, comparing buffers, and transforming data. Node.js provides a comprehensive set of methods for these operations, from simple byte access to complex multi-byte integer reads.

Think of Buffer operations like working with a specialized array where each element is a byte (0-255), but with additional superpowers for handling binary data patterns, different number formats, and efficient memory operations.

What You Need to Know First

Required reading (in order):

1. Buffer Basics: Working with Binary Data in Node.js - Understanding Buffer creation and fundamentals
2. Buffer Memory Management: Allocation, Pools, and Performance - How buffers are allocated

Helpful background:

• Basic understanding of binary and hexadecimal numbers
• Familiarity with JavaScript array operations (helpful but not required)

What We'll Cover in This Article

By the end of this guide, you'll master:

• Direct byte access and modification
• Reading and writing strings with different encodings
• Buffer slicing and creating views
• Copying data between buffers
• Comparing and searching buffers
• Filling buffers with patterns
• Working with multi-byte integers
• Buffer concatenation and manipulation

What We'll Explain Along the Way

These concepts will be explained with examples:

• The difference between views and copies (critical concept)
• Little-endian vs big-endian byte order (with visual diagrams)
• Integer types (signed vs unsigned, 8-bit, 16-bit, 32-bit)
• Buffer immutability vs mutability
• Index boundaries and safety

Basic Buffer Access: Reading and Writing Bytes

Direct Byte Access

Buffers behave like arrays where each element is a byte (0-255):

// Create a buffer with some data
const buffer = Buffer.from([72, 101, 108, 108, 111]); // "Hello"

// Read individual bytes
console.log(buffer[0]); // 72 ('H')
console.log(buffer[1]); // 101 ('e')
console.log(buffer[4]); // 111 ('o')

// Write individual bytes
buffer[0] = 74; // Change 'H' (72) to 'J' (74)
console.log(buffer.toString()); // "Jello"

// Check buffer length
console.log(buffer.length); // 5 bytes

// Access last byte
console.log(buffer[buffer.length - 1]); // 111 ('o')

Visual Representation:

Buffer: [72, 101, 108, 108, 111]
Index:   0    1    2    3    4
ASCII:   H    e    l    l    o

After buffer[0] = 74:

Buffer: [74, 101, 108, 108, 111]
Index:   0    1    2    3    4
ASCII:   J    e    l    l    o

Bounds Checking and Safety

const buffer = Buffer.from([1, 2, 3, 4, 5]);

// ✅ Reading within bounds
console.log(buffer[0]); // 1
console.log(buffer[4]); // 5

// ⚠️ Reading outside bounds (returns undefined)
console.log(buffer[5]); // undefined
console.log(buffer[10]); // undefined
console.log(buffer[-1]); // undefined

// ⚠️ Writing outside bounds (silently ignored)
buffer[10] = 99; // No error, but does nothing
console.log(buffer.length); // Still 5 (not expanded)
console.log(buffer[10]); // undefined

// Why?
// Buffers have FIXED size
// You cannot grow or shrink them
// Out-of-bounds operations are safe but ineffective

Checking Bounds Before Access:

function safeReadByte(buffer: Buffer, index: number): number | undefined {
  // Check if index is valid
  if (index < 0 || index >= buffer.length) {
    console.log(`Index ${index} out of bounds (length: ${buffer.length})`);
    return undefined;
  }
  return buffer[index];
}

const buf = Buffer.from([10, 20, 30]);
console.log(safeReadByte(buf, 1)); // 20
console.log(safeReadByte(buf, 5)); // undefined (with warning)

Iterating Over Buffer Bytes

const buffer = Buffer.from([72, 101, 108, 108, 111]);

// Method 1: Traditional for loop
console.log("Method 1 - for loop:");
for (let i = 0; i < buffer.length; i++) {
  console.log(
    `buffer[${i}] = ${buffer[i]} ('${String.fromCharCode(buffer[i])}')`
  );
}

// Method 2: for...of loop
console.log("\nMethod 2 - for...of:");
for (const byte of buffer) {
  console.log(`Byte: ${byte} ('${String.fromCharCode(byte)}')`);
}

// Method 3: forEach (Buffer extends Uint8Array)
console.log("\nMethod 3 - forEach:");
buffer.forEach((byte, index) => {
  console.log(`buffer[${index}] = ${byte}`);
});

// Method 4: Array methods (map, filter, reduce)
console.log("\nMethod 4 - Array methods:");
const doubled = Array.from(buffer).map((byte) => byte * 2);
console.log(doubled); // [144, 202, 216, 216, 222]

Buffer Properties and Length

buffer.length vs buffer.byteLength

const buffer = Buffer.from([1, 2, 3, 4, 5]);

// buffer.length: Number of bytes in the buffer
console.log(buffer.length); // 5

// buffer.byteLength: Same as buffer.length (for compatibility)
console.log(buffer.byteLength); // 5

// They're always equal for Buffers
console.log(buffer.length === buffer.byteLength); // true

// Note: buffer.buffer.byteLength is different (underlying ArrayBuffer size)
console.log(buffer.buffer.byteLength); // 8192 (if from pool)

String Length vs Buffer Length:

// ASCII string: 1 character = 1 byte
const ascii = "Hello";
const asciiBuffer = Buffer.from(ascii);
console.log(ascii.length); // 5 characters
console.log(asciiBuffer.length); // 5 bytes

// UTF-8 string: Multi-byte characters
const utf8 = "café"; // 'é' is 2 bytes in UTF-8
const utf8Buffer = Buffer.from(utf8);
console.log(utf8.length); // 4 characters
console.log(utf8Buffer.length); // 5 bytes

// Emoji string: Even more bytes
const emoji = "Hello 👋"; // Wave emoji is 4 bytes
const emojiBuffer = Buffer.from(emoji);
console.log(emoji.length); // 8 (JavaScript counts UTF-16 code units)
console.log(emojiBuffer.length); // 10 bytes in UTF-8

// Lesson: NEVER assume string.length === buffer.length

String Operations: Reading and Writing Text

buffer.toString() - Convert Buffer to String

const buffer = Buffer.from([72, 101, 108, 108, 111]);

// Default encoding: UTF-8
console.log(buffer.toString()); // "Hello"

// Specify encoding explicitly
console.log(buffer.toString("utf8")); // "Hello"

// Convert to hex representation
console.log(buffer.toString("hex")); // "48656c6c6f"

// Convert to base64
console.log(buffer.toString("base64")); // "SGVsbG8="

// Convert to binary string
console.log(buffer.toString("binary")); // "Hello"

Partial String Conversion:

const buffer = Buffer.from("Hello World");

// Extract portion of buffer as string
console.log(buffer.toString("utf8", 0, 5)); // "Hello"
console.log(buffer.toString("utf8", 6, 11)); // "World"
console.log(buffer.toString("utf8", 6)); // "World" (to end)

// Breakdown:
// toString(encoding, start, end)
// start: Starting byte index (inclusive)
// end: Ending byte index (exclusive)

// Visual:
// Buffer: H  e  l  l  o     W  o  r  l  d
// Index:  0  1  2  3  4  5  6  7  8  9  10
//        └─────────┘        └──────────┘
//         "Hello"              "World"

Different Encodings:

// Create buffer with multi-byte characters
const text = "café";
const buffer = Buffer.from(text, "utf8");

console.log(buffer); // <Buffer 63 61 66 c3 a9>
console.log(buffer.length); // 5 bytes

// Convert back with different encodings
console.log(buffer.toString("utf8")); // "café" (correct)
console.log(buffer.toString("ascii")); // "café" (wrong! ASCII can't handle 0xC3 0xA9)
console.log(buffer.toString("latin1")); // "café" (wrong! Latin-1 treats bytes individually)
console.log(buffer.toString("hex")); // "636166c3a9" (hex representation)

// Lesson: Always use the same encoding for encode/decode

buffer.write() - Write String to Buffer

// Create an empty buffer
const buffer = Buffer.alloc(20);

// Write a string
const bytesWritten = buffer.write("Hello");
console.log(`Wrote ${bytesWritten} bytes`); // 5
console.log(buffer.toString("utf8", 0, bytesWritten)); // "Hello"
console.log(buffer); // <Buffer 48 65 6c 6c 6f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

// Write with offset
buffer.write("World", 6); // Start at byte 6
console.log(buffer.toString("utf8", 0, 11)); // "Hello World"

// Write with length limit
const buf2 = Buffer.alloc(10);
buf2.write("Hello World", 0, 5); // Only write 5 bytes
console.log(buf2.toString()); // "Hello" (truncated)

Write with Different Encodings:

const buffer = Buffer.alloc(20);

// Write UTF-8 (default)
buffer.write("café", 0, "utf8");
console.log(buffer.slice(0, 5)); // <Buffer 63 61 66 c3 a9>

// Write hex string
buffer.write("48656c6c6f", 0, "hex");
console.log(buffer.toString("utf8", 0, 5)); // "Hello"

// Write base64
buffer.write("SGVsbG8=", 0, "base64");
console.log(buffer.toString("utf8", 0, 5)); // "Hello"

// Full signature:
// buffer.write(string, offset, length, encoding)

Handling Write Overflow:

const buffer = Buffer.alloc(5);

// Try to write more than buffer can hold
const result = buffer.write("Hello World");

console.log(`Bytes written: ${result}`); // 5 (only what fits)
console.log(buffer.toString()); // "Hello" (rest truncated)

// Safe writing pattern:
function safeWrite(buffer: Buffer, text: string, offset = 0): boolean {
  const bytesNeeded = Buffer.byteLength(text);
  const spaceAvailable = buffer.length - offset;

  if (bytesNeeded > spaceAvailable) {
    console.log(
      `⚠️  Not enough space! Need ${bytesNeeded}, have ${spaceAvailable}`
    );
    return false;
  }

  buffer.write(text, offset);
  return true;
}

const buf = Buffer.alloc(10);
safeWrite(buf, "Hi", 0); // ✓ Success
safeWrite(buf, "Hello World", 0); // ✗ Failure (warning shown)

Buffer Slicing and Views

buffer.slice() - Create View (Not a Copy!)

Critical Concept: slice() creates a view over the original buffer, not a copy. Changes to the slice affect the original!

const original = Buffer.from("Hello World");

// Create a slice (view)
const slice = original.slice(0, 5); // "Hello"

console.log(slice.toString()); // "Hello"

// Modify the slice
slice[0] = 74; // Change 'H' (72) to 'J' (74)

// Original is also modified!
console.log(slice.toString()); // "Jello"
console.log(original.toString()); // "Jello World" (changed!)

// Why? slice() creates a VIEW, not a copy
console.log(slice.buffer === original.buffer); // true (same underlying memory)

Visual Explanation:

Original Buffer:
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│H│e│l│l│o│ │W│o│r│l│d│
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
 0 1 2 3 4 5 6 7 8 9 10

Slice (view of bytes 0-4):
┌─┬─┬─┬─┬─┐
│H│e│l│l│o│ ← Points to original buffer's memory
└─┴─┴─┴─┴─┘

After slice[0] = 74:
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│J│e│l│l│o│ │W│o│r│l│d│ ← Original also changed
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘

Creating True Copies:

const original = Buffer.from("Hello World");

// Method 1: Buffer.from() - creates a copy
const copy1 = Buffer.from(original.slice(0, 5));
copy1[0] = 74;
console.log(copy1.toString()); // "Jello"
console.log(original.toString()); // "Hello World" (unchanged)

// Method 2: Allocate new buffer and copy
const copy2 = Buffer.alloc(5);
original.copy(copy2, 0, 0, 5);
copy2[0] = 74;
console.log(copy2.toString()); // "Jello"
console.log(original.toString()); // "Hello World" (unchanged)

// Verify they're separate
console.log(copy1.buffer === original.buffer); // false
console.log(copy2.buffer === original.buffer); // false

buffer.subarray() - Explicit View Creation

subarray() is the modern, explicit way to create views (same as slice()):

const original = Buffer.from("Hello World");

// subarray() and slice() are identical for Buffers
const view1 = original.slice(0, 5);
const view2 = original.subarray(0, 5);

console.log(view1.toString()); // "Hello"
console.log(view2.toString()); // "Hello"

// Both are views
view1[0] = 74;
console.log(original.toString()); // "Jello World"

view2[1] = 65; // 'A'
console.log(original.toString()); // "JAllo World"

// Recommendation: Use subarray() for clarity
// It makes it obvious you're creating a view, not a copy

Practical Slicing Examples

// Example 1: Extract header from binary data
const fileData = Buffer.from([
  0xff,
  0xd8,
  0xff,
  0xe0, // JPEG header
  0x00,
  0x10, // Length
  // ... rest of file
  0x01,
  0x02,
  0x03,
  0x04,
]);

const header = fileData.subarray(0, 4);
console.log(header); // <Buffer ff d8 ff e0>

// Check if it's a JPEG
if (header[0] === 0xff && header[1] === 0xd8) {
  console.log("This is a JPEG file!");
}

// Example 2: Process chunks
const data = Buffer.from("AAAABBBBCCCCDDDD");
const chunkSize = 4;

for (let i = 0; i < data.length; i += chunkSize) {
  const chunk = data.subarray(i, i + chunkSize);
  console.log(`Chunk ${i / chunkSize}: ${chunk.toString()}`);
}
// Output:
// Chunk 0: AAAA
// Chunk 1: BBBB
// Chunk 2: CCCC
// Chunk 3: DDDD

// Example 3: Extract specific fields from binary protocol
const packet = Buffer.from([
  0x01,
  0x02, // Type (2 bytes)
  0x00,
  0x0a, // Length (2 bytes)
  0x48,
  0x65,
  0x6c,
  0x6c,
  0x6f, // Data ("Hello")
  0x00,
  0x00,
  0x00,
  0x00,
  0x00, // Padding
]);

const type = packet.subarray(0, 2);
const length = packet.subarray(2, 4);
const payload = packet.subarray(4, 4 + packet.readUInt16BE(2));

console.log("Type:", type.toString("hex")); // "0102"
console.log("Length:", length.readUInt16BE(0)); // 10
console.log("Data:", payload.toString()); // "Hello"

Buffer Copying

buffer.copy() - Copy Data Between Buffers

const source = Buffer.from("Hello World");
const target = Buffer.alloc(20);

// Copy entire source to target
source.copy(target);
console.log(target.toString()); // "Hello World" + zeros

// Full signature:
// source.copy(target, targetStart, sourceStart, sourceEnd)

// Example: Copy partial data
const source2 = Buffer.from("Hello World");
const target2 = Buffer.alloc(10);

// Copy "World" (bytes 6-11) to start of target2
source2.copy(target2, 0, 6, 11);
console.log(target2.toString("utf8", 0, 5)); // "World"

Visual Explanation:

source.copy(target, 2, 0, 5):

Source:
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│H│e│l│l│o│ │W│o│r│l│d│
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
 └─────────┘ Copy bytes 0-5
     ↓
Target:
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│0│0│H│e│l│l│o│0│0│0│
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
     ↑ Paste at index 2

Copy with Offset:

const source = Buffer.from("ABCD");
const target = Buffer.alloc(10).fill(45); // Fill with '-' (45)

// Copy to different positions
source.copy(target, 0); // Copy at start
source.copy(target, 5); // Copy at middle

console.log(target.toString());
// "ABCD-ABCD-" (two copies of source)

// Breakdown:
// Position 0-3: "ABCD"
// Position 4: "-"
// Position 5-8: "ABCD"
// Position 9: "-"

Handling Overflow:

const source = Buffer.from("Hello World"); // 11 bytes
const target = Buffer.alloc(5); // Only 5 bytes

// Copy will truncate to fit target
const bytesCopied = source.copy(target);

console.log(`Copied ${bytesCopied} bytes`); // 5
console.log(target.toString()); // "Hello" (truncated)

// Safe copying pattern:
function safeCopy(source: Buffer, target: Buffer, targetOffset = 0): number {
  const spaceAvailable = target.length - targetOffset;
  const bytesToCopy = Math.min(source.length, spaceAvailable);

  if (bytesToCopy < source.length) {
    console.log(
      `⚠️  Truncating: ${source.length} bytes to ${bytesToCopy} bytes`
    );
  }

  return source.copy(target, targetOffset, 0, bytesToCopy);
}

Buffer.concat() - Combine Multiple Buffers

const buf1 = Buffer.from("Hello ");
const buf2 = Buffer.from("World");
const buf3 = Buffer.from("!");

// Concatenate buffers
const combined = Buffer.concat([buf1, buf2, buf3]);

console.log(combined.toString()); // "Hello World!"
console.log(combined.length); // 12 bytes

// How it works internally:
// 1. Calculate total length: 6 + 5 + 1 = 12
// 2. Allocate new buffer of 12 bytes
// 3. Copy buf1 to position 0
// 4. Copy buf2 to position 6
// 5. Copy buf3 to position 11

Specify Total Length:

const buf1 = Buffer.from("Hello");
const buf2 = Buffer.from("World");

// Specify total length (useful if you know it)
const combined1 = Buffer.concat([buf1, buf2], 10);
console.log(combined1.toString()); // "HelloWorld"

// If specified length is larger, fills with zeros
const combined2 = Buffer.concat([buf1, buf2], 20);
console.log(combined2.length); // 20
console.log(combined2.toString()); // "HelloWorld" + zeros

// If specified length is smaller, truncates
const combined3 = Buffer.concat([buf1, buf2], 7);
console.log(combined3.toString()); // "HelloWo" (truncated)

Practical Example: Building Protocol Messages:

// Build a simple protocol message
function buildMessage(type: number, data: string): Buffer {
  const header = Buffer.alloc(4);
  header.writeUInt16BE(type, 0); // Message type (2 bytes)
  header.writeUInt16BE(data.length, 2); // Data length (2 bytes)

  const payload = Buffer.from(data);

  return Buffer.concat([header, payload]);
}

const message = buildMessage(1, "Hello");
console.log(message);
// <Buffer 00 01 00 05 48 65 6c 6c 6f>
// [type][len ][H][e][l][l][o]

// Parse it back
function parseMessage(buffer: Buffer) {
  const type = buffer.readUInt16BE(0);
  const length = buffer.readUInt16BE(2);
  const data = buffer.toString("utf8", 4, 4 + length);

  return { type, length, data };
}

const parsed = parseMessage(message);
console.log(parsed); // { type: 1, length: 5, data: 'Hello' }

Buffer Comparison and Searching

buffer.equals() - Compare Entire Buffers

const buf1 = Buffer.from("Hello");
const buf2 = Buffer.from("Hello");
const buf3 = Buffer.from("World");

// Compare buffers
console.log(buf1.equals(buf2)); // true (same content)
console.log(buf1.equals(buf3)); // false (different content)

// Even if same reference
console.log(buf1.equals(buf1)); // true

// Note: This compares CONTENT, not reference
console.log(buf1 === buf2); // false (different objects)
console.log(buf1.equals(buf2)); // true (same content)

Case-Sensitive Comparison:

const buf1 = Buffer.from("Hello");
const buf2 = Buffer.from("hello");

console.log(buf1.equals(buf2)); // false (case matters!)

// For case-insensitive comparison:
const lower1 = Buffer.from(buf1.toString().toLowerCase());
const lower2 = Buffer.from(buf2.toString().toLowerCase());
console.log(lower1.equals(lower2)); // true

buffer.compare() - Lexicographic Comparison

const buf1 = Buffer.from("ABC");
const buf2 = Buffer.from("BCD");
const buf3 = Buffer.from("ABC");

// compare() returns:
// - Negative if buf1 < buf2
// - Zero if buf1 === buf2
// - Positive if buf1 > buf2

console.log(buf1.compare(buf2)); // -1 (buf1 < buf2)
console.log(buf2.compare(buf1)); // 1 (buf2 > buf1)
console.log(buf1.compare(buf3)); // 0 (buf1 === buf3)

// Compare portions
const buf4 = Buffer.from("Hello World");
const buf5 = Buffer.from("World");

// Compare "World" part of buf4 with buf5
const result = buf4.compare(buf5, 0, 5, 6, 11);
console.log(result); // 0 (they match)

// Signature:
// buf.compare(target, targetStart, targetEnd, sourceStart, sourceEnd)

Sorting Buffers:

const buffers = [
  Buffer.from("zebra"),
  Buffer.from("apple"),
  Buffer.from("mango"),
  Buffer.from("banana"),
];

// Sort using compare()
buffers.sort((a, b) => a.compare(b));

console.log(buffers.map((buf) => buf.toString()));
// ["apple", "banana", "mango", "zebra"]

buffer.indexOf() - Search for Value

const buffer = Buffer.from("Hello World Hello");

// Find first occurrence
console.log(buffer.indexOf("o")); // 4 (first 'o' in "Hello")

// Find with starting offset
console.log(buffer.indexOf("o", 5)); // 7 (next 'o' in "World")

// Find using buffer
const search = Buffer.from("World");
console.log(buffer.indexOf(search)); // 6

// Find using byte value
console.log(buffer.indexOf(111)); // 4 ('o' = 111)

// Not found returns -1
console.log(buffer.indexOf("xyz")); // -1

Find All Occurrences:

function findAll(buffer: Buffer, search: string | Buffer): number[] {
  const positions: number[] = [];
  let index = 0;

  while (true) {
    index = buffer.indexOf(search, index);
    if (index === -1) break;

    positions.push(index);
    index++; // Move past this occurrence
  }

  return positions;
}

const buffer = Buffer.from("Hello World Hello Universe Hello");
const positions = findAll(buffer, "Hello");
console.log(positions); // [0, 12, 27]

// Verify:
positions.forEach((pos) => {
  console.log(buffer.toString("utf8", pos, pos + 5)); // "Hello"
});

buffer.lastIndexOf() - Search Backwards

const buffer = Buffer.from("Hello World Hello");

// Find last occurrence
console.log(buffer.lastIndexOf("Hello")); // 12 (last "Hello")

// Search backwards from position
console.log(buffer.lastIndexOf("Hello", 10)); // 0 (search from pos 10 backwards)

// Not found
console.log(buffer.lastIndexOf("xyz")); // -1

buffer.includes() - Check if Contains Value

const buffer = Buffer.from("Hello World");

// Check if contains string
console.log(buffer.includes("World")); // true
console.log(buffer.includes("xyz")); // false

// Check if contains buffer
const search = Buffer.from("World");
console.log(buffer.includes(search)); // true

// Check if contains byte value
console.log(buffer.includes(111)); // true ('o' = 111)
console.log(buffer.includes(255)); // false

// With starting offset
console.log(buffer.includes("o", 5)); // true (second 'o' in "World")
console.log(buffer.includes("o", 8)); // false (no 'o' after position 8)

Buffer Filling

buffer.fill() - Fill Buffer with Value

// Fill entire buffer with a value
const buf1 = Buffer.alloc(10);
buf1.fill(65); // Fill with 'A' (ASCII 65)
console.log(buf1.toString()); // "AAAAAAAAAA"

// Fill with string
const buf2 = Buffer.alloc(10);
buf2.fill("abc"); // Pattern repeats
console.log(buf2.toString()); // "abcabcabca"

// Fill with buffer
const buf3 = Buffer.alloc(10);
const pattern = Buffer.from([1, 2, 3]);
buf3.fill(pattern);
console.log(buf3); // <Buffer 01 02 03 01 02 03 01 02 03 01>

// Fill portion of buffer
const buf4 = Buffer.alloc(10).fill(45); // All '-'
buf4.fill("X", 2, 8); // Fill bytes 2-7 with 'X'
console.log(buf4.toString()); // "--XXXXXX--"

Visual Representation:

buf.fill("ABC", 2, 8):

Before (all zeros):
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│0│0│0│0│0│0│0│0│0│0│
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
 0 1 2 3 4 5 6 7 8 9

After fill("ABC", 2, 8):
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│0│0│A│B│C│A│B│C│0│0│
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
     └─────────┘ Filled region

Practical Fill Patterns:

// Clear buffer (zero out)
function clearBuffer(buffer: Buffer): void {
  buffer.fill(0);
}

// Initialize with pattern
function initializeBuffer(size: number, pattern: string): Buffer {
  const buffer = Buffer.alloc(size);
  buffer.fill(pattern);
  return buffer;
}

// Create delimiter-separated buffer
function createDelimitedBuffer(): Buffer {
  const buffer = Buffer.alloc(20).fill(45); // Fill with '-'
  buffer.write("DATA", 0);
  buffer.write("DATA", 8);
  buffer.write("DATA", 16);
  return buffer;
}

console.log(createDelimitedBuffer().toString());
// "DATA----DATA----DATA"

Reading and Writing Multi-Byte Integers

Buffers provide methods to read and write integers of different sizes (8-bit, 16-bit, 32-bit) and byte orders (little-endian, big-endian).

Understanding Endianness

Endianness determines the order in which bytes are stored for multi-byte values.

Number: 0x12345678 (4 bytes)

Big-Endian (BE): Most significant byte first
Memory: [0x12][0x34][0x56][0x78]
         ↑ highest byte first
Used by: Network protocols, Java

Little-Endian (LE): Least significant byte first
Memory: [0x78][0x56][0x34][0x12]
         ↑ lowest byte first
Used by: x86/x64 CPUs, Windows

8-Bit Integer Operations (1 byte)

const buffer = Buffer.alloc(10);

// Write unsigned 8-bit integer (0-255)
buffer.writeUInt8(255, 0);
buffer.writeUInt8(128, 1);
buffer.writeUInt8(0, 2);

console.log(buffer.slice(0, 3)); // <Buffer ff 80 00>

// Read unsigned 8-bit integer
console.log(buffer.readUInt8(0)); // 255
console.log(buffer.readUInt8(1)); // 128
console.log(buffer.readUInt8(2)); // 0

// Write signed 8-bit integer (-128 to 127)
buffer.writeInt8(-1, 3);
buffer.writeInt8(-128, 4);
buffer.writeInt8(127, 5);

console.log(buffer.slice(3, 6)); // <Buffer ff 80 7f>

// Read signed 8-bit integer
console.log(buffer.readInt8(3)); // -1
console.log(buffer.readInt8(4)); // -128
console.log(buffer.readInt8(5)); // 127

Signed vs Unsigned:

const buffer = Buffer.from([0xff, 0x80, 0x7f]);

// As unsigned (0-255)
console.log(buffer.readUInt8(0)); // 255
console.log(buffer.readUInt8(1)); // 128
console.log(buffer.readUInt8(2)); // 127

// As signed (-128 to 127)
console.log(buffer.readInt8(0)); // -1 (0xFF = -1 in two's complement)
console.log(buffer.readInt8(1)); // -128 (0x80 = -128)
console.log(buffer.readInt8(2)); // 127 (0x7F = 127)

// Why different?
// 0xFF in binary: 11111111
// Unsigned: 255
// Signed: -1 (two's complement)

16-Bit Integer Operations (2 bytes)

const buffer = Buffer.alloc(10);

// Write 16-bit unsigned integer - Little Endian
buffer.writeUInt16LE(1000, 0);
console.log(buffer.slice(0, 2)); // <Buffer e8 03>
// 1000 = 0x03E8
// Little-endian: [0xE8, 0x03] (low byte first)

// Write 16-bit unsigned integer - Big Endian
buffer.writeUInt16BE(1000, 2);
console.log(buffer.slice(2, 4)); // <Buffer 03 e8>
// Big-endian: [0x03, 0xE8] (high byte first)

// Read back
console.log(buffer.readUInt16LE(0)); // 1000
console.log(buffer.readUInt16BE(2)); // 1000

// Signed 16-bit integers
buffer.writeInt16LE(-1000, 4);
buffer.writeInt16BE(-1000, 6);

console.log(buffer.readInt16LE(4)); // -1000
console.log(buffer.readInt16BE(6)); // -1000

Visual Comparison:

Value: 1000 (0x03E8 in hex)

Little-Endian (LE):
┌────┬────┐
│ E8 │ 03 │  Low byte first
└────┴────┘
  ↑    ↑
 Low  High

Big-Endian (BE):
┌────┬────┐
│ 03 │ E8 │  High byte first
└────┴────┘
  ↑    ↑
High  Low

Reading with wrong endianness:
writeUInt16LE(1000) = [0xE8, 0x03]
readUInt16BE()      = 0xE803 = 59395 (WRONG!)

32-Bit Integer Operations (4 bytes)

const buffer = Buffer.alloc(20);

// Write 32-bit unsigned integer
buffer.writeUInt32LE(1000000, 0);
buffer.writeUInt32BE(1000000, 4);

console.log(buffer.slice(0, 4)); // <Buffer 40 42 0f 00> (LE)
console.log(buffer.slice(4, 8)); // <Buffer 00 0f 42 40> (BE)

// Read back
console.log(buffer.readUInt32LE(0)); // 1000000
console.log(buffer.readUInt32BE(4)); // 1000000

// Signed 32-bit integers
buffer.writeInt32LE(-1000000, 8);
buffer.writeInt32BE(-1000000, 12);

console.log(buffer.readInt32LE(8)); // -1000000
console.log(buffer.readInt32BE(12)); // -1000000

// Maximum values
const maxInt32 = 2147483647; // 2^31 - 1
const maxUInt32 = 4294967295; // 2^32 - 1

buffer.writeInt32LE(maxInt32, 16);
console.log(buffer.readInt32LE(16)); // 2147483647

buffer.writeUInt32LE(maxUInt32, 16);
console.log(buffer.readUInt32LE(16)); // 4294967295

Floating-Point Operations

const buffer = Buffer.alloc(20);

// Write 32-bit float (single precision)
buffer.writeFloatLE(3.14159, 0);
buffer.writeFloatBE(3.14159, 4);

console.log(buffer.readFloatLE(0)); // 3.1415927410125732 (slight precision loss)
console.log(buffer.readFloatBE(4)); // 3.1415927410125732

// Write 64-bit double (double precision)
buffer.writeDoubleLE(3.14159265359, 8);
buffer.writeDoubleBE(3.14159265359, 16);

console.log(buffer.readDoubleLE(8)); // 3.14159265359 (exact)
console.log(buffer.readDoubleBE(16)); // 3.14159265359

// Why use doubles?
const preciseValue = 1234567890.123456789;

buffer.writeFloatLE(preciseValue, 0);
buffer.writeDoubleLE(preciseValue, 4);

console.log("Float:", buffer.readFloatLE(0)); // 1234567936 (precision lost!)
console.log("Double:", buffer.readDoubleLE(4)); // 1234567890.123457 (much better)

Practical Example: Binary File Format

// Create a simple binary file format
interface FileHeader {
  magic: number; // 4 bytes: Magic number (0x4D59464D = "MYFM")
  version: number; // 2 bytes: Version
  flags: number; // 2 bytes: Flags
  timestamp: number; // 4 bytes: Unix timestamp
  dataLength: number; // 4 bytes: Data length
}

function writeHeader(header: FileHeader): Buffer {
  const buffer = Buffer.alloc(16); // 4+2+2+4+4 = 16 bytes

  buffer.writeUInt32BE(header.magic, 0); // Magic at 0
  buffer.writeUInt16BE(header.version, 4); // Version at 4
  buffer.writeUInt16BE(header.flags, 6); // Flags at 6
  buffer.writeUInt32BE(header.timestamp, 8); // Timestamp at 8
  buffer.writeUInt32BE(header.dataLength, 12); // Length at 12

  return buffer;
}

function readHeader(buffer: Buffer): FileHeader {
  return {
    magic: buffer.readUInt32BE(0),
    version: buffer.readUInt16BE(4),
    flags: buffer.readUInt16BE(6),
    timestamp: buffer.readUInt32BE(8),
    dataLength: buffer.readUInt32BE(12),
  };
}

// Usage
const header: FileHeader = {
  magic: 0x4d59464d, // "MYFM"
  version: 1,
  flags: 0x0003,
  timestamp: Math.floor(Date.now() / 1000),
  dataLength: 1024,
};

const headerBuffer = writeHeader(header);
console.log(headerBuffer);
// <Buffer 4d 59 46 4d 00 01 00 03 ...>

const parsedHeader = readHeader(headerBuffer);
console.log(parsedHeader);
// { magic: 1297371981, version: 1, flags: 3, ... }

// Verify magic number
const magicString = String.fromCharCode(
  (parsedHeader.magic >> 24) & 0xff,
  (parsedHeader.magic >> 16) & 0xff,
  (parsedHeader.magic >> 8) & 0xff,
  parsedHeader.magic & 0xff
);
console.log("Magic:", magicString); // "MYFM"

Buffer Swapping Operations

Node.js provides methods to swap byte order for 16-bit, 32-bit, and 64-bit values:

// 16-bit swap
const buf16 = Buffer.from([0x01, 0x02, 0x03, 0x04]);
console.log(buf16); // <Buffer 01 02 03 04>

buf16.swap16(); // Swap every 2 bytes
console.log(buf16); // <Buffer 02 01 04 03>

// 32-bit swap
const buf32 = Buffer.from([0x01, 0x02, 0x03, 0x04]);
console.log(buf32); // <Buffer 01 02 03 04>

buf32.swap32(); // Swap every 4 bytes
console.log(buf32); // <Buffer 04 03 02 01>

// 64-bit swap
const buf64 = Buffer.from([0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08]);
console.log(buf64); // <Buffer 01 02 03 04 05 06 07 08>

buf64.swap64(); // Swap every 8 bytes
console.log(buf64); // <Buffer 08 07 06 05 04 03 02 01>

// Use case: Converting endianness
const value = 0x12345678;
const buffer = Buffer.alloc(4);
buffer.writeUInt32BE(value, 0); // Write as big-endian

console.log(buffer); // <Buffer 12 34 56 78>

buffer.swap32(); // Convert to little-endian
console.log(buffer); // <Buffer 78 56 34 12>
console.log(buffer.readUInt32LE(0)); // 305419896 (0x12345678)

Common Pitfalls and Solutions

Pitfall 1: Confusing slice() with Copy

// ❌ Problem: Thinking slice() creates a copy
const original = Buffer.from("Hello");
const slice = original.slice(0, 3);
slice[0] = 88; // Change 'H' to 'X'

console.log(original.toString()); // "Xello" (original changed!)

// ✅ Solution: Use Buffer.from() for true copy
const original2 = Buffer.from("Hello");
const copy = Buffer.from(original2.slice(0, 3));
copy[0] = 88;

console.log(original2.toString()); // "Hello" (unchanged)
console.log(copy.toString()); // "Xel"

Pitfall 2: Wrong Endianness

// ❌ Problem: Reading with wrong endianness
const buffer = Buffer.alloc(4);
buffer.writeUInt32LE(1000, 0); // Write little-endian

const wrong = buffer.readUInt32BE(0); // Read big-endian
console.log(wrong); // 3925868544 (completely wrong!)

const correct = buffer.readUInt32LE(0); // Read little-endian
console.log(correct); // 1000 (correct)

// ✅ Solution: Always match read/write endianness
// Convention: Use BE for network protocols, LE for file formats (document it!)

Pitfall 3: Buffer Overflow on Write

// ❌ Problem: Writing beyond buffer bounds
const buffer = Buffer.alloc(4);
buffer.writeUInt32LE(1000, 0); // OK
buffer.writeUInt32LE(2000, 2); // DANGER: Writes bytes 2-5, but buffer is only 4 bytes!

// This will write partial data and may corrupt adjacent memory
console.log(buffer); // <Buffer e8 03 d0 07>
// Only bytes 2-3 are written (partial 32-bit value)

// ✅ Solution: Always check bounds
function safeWriteUInt32(
  buffer: Buffer,
  value: number,
  offset: number
): boolean {
  if (offset + 4 > buffer.length) {
    console.log("⚠️  Not enough space for UInt32");
    return false;
  }
  buffer.writeUInt32LE(value, offset);
  return true;
}

Pitfall 4: Forgetting toString() Encoding

// ❌ Problem: Implicit UTF-8 conversion loses data
const buffer = Buffer.from([0xff, 0xfe, 0xfd]);
console.log(buffer.toString()); // "���" (UTF-8 tries to decode as characters)

// ✅ Solution: Use appropriate encoding
console.log(buffer.toString("hex")); // "fffefd" (correct for binary data)
console.log(buffer.toString("base64")); // "//79" (correct for binary data)

// For raw binary display:
console.log(buffer); // <Buffer ff fe fd> (best for debugging)

Pitfall 5: Comparing Buffers with ===

// ❌ Problem: Using === compares references, not content
const buf1 = Buffer.from("Hello");
const buf2 = Buffer.from("Hello");

console.log(buf1 === buf2); // false (different objects)
console.log(buf1 == buf2); // false (still different objects)

// ✅ Solution: Use equals() for content comparison
console.log(buf1.equals(buf2)); // true (same content)

// Or compare() for ordering
console.log(buf1.compare(buf2)); // 0 (equal)

Pitfall 6: Modifying Shared Buffers

// ❌ Problem: Multiple references to same buffer
const original = Buffer.from("Hello");
const ref1 = original;
const ref2 = original.slice(0, 5); // Still a view!

ref1[0] = 88;
console.log(ref2.toString()); // "Xello" (all references affected)

// ✅ Solution: Be aware of references and use copies when needed
const original2 = Buffer.from("Hello");
const copy1 = Buffer.from(original2); // True copy
const copy2 = Buffer.from(original2); // Another copy

copy1[0] = 88;
console.log(original2.toString()); // "Hello" (unchanged)
console.log(copy2.toString()); // "Hello" (unchanged)

Summary: Key Takeaways

Basic Operations:

• Direct byte access: Use buffer[index] to read/write individual bytes (0-255)
• String conversion: toString(encoding) converts to string, write() writes string to buffer
• Length properties: buffer.length is byte count (not character count)

Slicing and Copying:

• slice() creates VIEWS - modifications affect the original buffer
• Use Buffer.from() to create true copies of slices
• copy() explicitly copies data between buffers
• concat() combines multiple buffers into one new buffer

Comparison and Search:

• equals() - Compare buffer contents (returns boolean)
• compare() - Lexicographic comparison (returns -1, 0, or 1)
• indexOf() - Find first occurrence
• lastIndexOf() - Find last occurrence
• includes() - Check if contains value

Filling:

• fill(value) - Fill entire buffer or portion with value
• Patterns repeat if value is shorter than fill region

Multi-Byte Integers:

• 8-bit: readUInt8, writeUInt8, readInt8, writeInt8
• 16-bit: readUInt16LE/BE, writeUInt16LE/BE, readInt16LE/BE, writeInt16LE/BE
• 32-bit: readUInt32LE/BE, writeUInt32LE/BE, readInt32LE/BE, writeInt32LE/BE
• Floats: readFloatLE/BE, writeFloatLE/BE, readDoubleLE/BE, writeDoubleLE/BE
• Endianness matters: LE (little-endian) for x86, BE (big-endian) for network protocols

Common Mistakes:

• Treating slice() as copy (it's a view!)
• Using wrong endianness for read/write
• Writing beyond buffer bounds
• Comparing buffers with === instead of equals()
• Forgetting that string length ≠ buffer length for UTF-8

What's Next?

Now that you've mastered Buffer operations, continue with:

• Buffer Encoding and Conversion - Deep dive into character encodings (UTF-8, UTF-16, Base64, Hex), converting between formats, and handling international text