Architecting High-Performance File Uploads: Chunking, Streams, and S3-Compatible Storage

An engineering deep dive into building ultra-fast, resilient file upload infrastructure: comparing multipart chunking, direct-to-R2/S3 presigned URLs, streaming pipelines, and token bucket rate limiters.

By X02 Team · 2026-08-22 · Performance, Architecture, JavaScript, Storage, S3, Cloudflare R2

Handling file uploads in modern web applications sounds deceptively simple until you encounter real-world constraints: users uploading 500MB video files over unstable 4G mobile connections, sudden spikes in concurrent traffic overwhelming Node.js memory heaps, and server timeouts dropping uploads at 99% progress.

At X02, our ingestion infrastructure processes millions of uploads globally with 99.99% reliability. In this article, we dissect the architectural principles behind high-throughput upload systems, compare streaming proxy architectures against direct presigned S3/R2 uploads, and examine resilient client-side retry strategies.

1. Upload Architectures: Monolithic vs. Presigned Direct-to-Storage

When designing a cloud file uploader, there are two primary architectural paradigms:

Architecture Diagram

How X02 Implements a Hybrid High-Performance Ingestion Layer

X02 combines the best of both approaches:

Every upload streams through Busboy/Multer straight into Cloudflare R2, so the server never holds a whole file in memory. Images are copied to disk as they pass through, and X02-Vision AI scans them right after the upload completes.
Files over 5MB (up to 512MB on Pro) go to R2 as Chunked Multipart Uploads: the stream is cut into 5MB parts, two in flight at a time, and R2 joins them into one object when the last part lands.

2. Chunked Multipart Uploads & Resumability

Uploading a 300MB archive as a single monolithic HTTP request is fragile. If the user's connection drops for 200ms at megabyte 290, the entire 290MB is lost and must restart from zero.

Chunked Upload Protocol Mechanics:

Initialize: Client calculates total file size and hashes the file's first block.
Partition: The file is sliced into uniform chunks using File.prototype.slice() in the browser.
Concurrent Upload: Up to 3-5 chunks are transmitted simultaneously.
Idempotent Retry: If Chunk #4 fails due to network failure, only Chunk #4 retries with exponential backoff.
Complete: When all parts are confirmed, an atomic commit command stitches the parts into a single object.

// Browser client-side chunked upload algorithm
async function uploadLargeFileInChunks(file, uploadEndpoint, apiKey) {
  const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB chunks
  const totalChunks = Math.ceil(file.size / CHUNK_SIZE);
  const uploadId = crypto.randomUUID();

  console.log(`Starting chunked upload: ${totalChunks} chunks for ${file.name}`);

  for (let index = 0; index < totalChunks; index++) {
    const start = index * CHUNK_SIZE;
    const end = Math.min(start + CHUNK_SIZE, file.size);
    const chunkBlob = file.slice(start, end);

    const formData = new FormData();
    formData.append("chunk", chunkBlob);
    formData.append("uploadId", uploadId);
    formData.append("chunkIndex", index.toString());
    formData.append("totalChunks", totalChunks.toString());
    formData.append("filename", file.name);

    let retries = 3;
    while (retries > 0) {
      try {
        const res = await fetch(`${uploadEndpoint}/chunk`, {
          method: "POST",
          headers: { "x-api-key": apiKey },
          body: formData
        });
        if (!res.ok) throw new Error(`Chunk ${index} failed`);
        break; // Success
      } catch (err) {
        retries--;
        if (retries === 0) throw err;
        await new Promise(r => setTimeout(r, 1000 * Math.pow(2, 3 - retries)));
      }
    }
  }

  // Finalize upload
  const finishRes = await fetch(`${uploadEndpoint}/finalize`, {
    method: "POST",
    headers: { "Content-Type": "application/json", "x-api-key": apiKey },
    body: JSON.stringify({ uploadId, filename: file.name, totalChunks })
  });

  return finishRes.json();
}

3. Backpressure & Memory-Safe Node.js Streams

A frequent cause of production Out-Of-Memory (OOM) crashes in Node.js backends is buffering entire uploaded files in RAM:

// Anti-Pattern: Buffering entire 200MB file into V8 heap RAM
const buffer = await req.file.buffer; // 200 concurrent users will crash the process (40GB RAM)

The Solution: Direct Stream Piping with Backpressure

By piping the incoming request stream directly into the S3 multipart upload stream, memory consumption stays flat regardless of file size: the SDK only holds the parts it is sending, about (queueSize + 1) × partSize. With the settings below that is roughly 25MB per upload, whether the file is 50MB or 5GB. (X02 runs two parts in flight, about 15MB of buffers, around 20–30MB per upload in practice.)

// High Performance: Stream piping with minimal memory overhead
import { PassThrough } from "stream";
import { Upload } from "@aws-sdk/lib-storage";
import { S3Client } from "@aws-sdk/client-s3";

async function streamUploadToR2(reqStream: NodeJS.ReadableStream, key: string, mime: string) {
  const passThrough = new PassThrough();
  reqStream.pipe(passThrough);

  const parallelUploads3 = new Upload({
    client: s3Client,
    params: {
      Bucket: process.env.R2_BUCKET_NAME,
      Key: key,
      Body: passThrough,
      ContentType: mime,
    },
    queueSize: 4,
    partSize: 5 * 1024 * 1024,
    leavePartsOnError: false,
  });

  parallelUploads3.on("httpUploadProgress", (progress) => {
    console.log(`Piped ${progress.loaded} bytes to storage.`);
  });

  return parallelUploads3.done();
}

4. Token Bucket Rate Limiting for Upload Protection

To prevent Denial of Service (DoS) and API abuse while maintaining smooth burst capabilities for legitimate users, X02 enforces a Redis-backed Token Bucket algorithm.

// Token Bucket rate limiter implementation for uploads
export class TokenBucketLimiter {
  private capacity: number;
  private refillRate: number; // Tokens per second

  constructor(capacity = 50, refillRate = 2) {
    this.capacity = capacity;
    this.refillRate = refillRate;
  }

  async consume(userId: string, tokensRequested = 1): Promise<{ allowed: boolean; remaining: number }> {
    const key = `rate:upload:${userId}`;
    const now = Date.now();
    
    // In production, execute atomic Redis Lua script
    const data = await redis.hgetall(key);
    let tokens = data.tokens ? parseFloat(data.tokens) : this.capacity;
    let lastRefill = data.lastRefill ? parseInt(data.lastRefill, 10) : now;

    // Calculate replenishment
    const elapsedSeconds = (now - lastRefill) / 1000;
    tokens = Math.min(this.capacity, tokens + elapsedSeconds * this.refillRate);

    if (tokens >= tokensRequested) {
      tokens -= tokensRequested;
      await redis.hmset(key, { tokens: tokens.toString(), lastRefill: now.toString() });
      return { allowed: true, remaining: Math.floor(tokens) };
    }

    return { allowed: false, remaining: Math.floor(tokens) };
  }
}

Summary

Scaling file uploads to millions of daily requests requires moving beyond monolithic synchronous handlers. By employing direct stream piping, chunked client-side resumability, and hybrid storage routing, you can build a file ingestion layer that is both resilient against network degradation and infinitely horizontally scalable.

Read this article on X02