webdevelop pro

Enterprise-grade file uploads, part 1: upload directly to S3

First article in a series on what "enterprise-grade" actually means for user file uploads — beyond the AWS-tutorial happy path.

Every product eventually has to accept files from users. Avatar pictures, KYC documents, signed contracts, transaction proofs, video clips. The naive way — "POST a multipart form to your API, the API writes to disk or to S3 — works on day one and explodes around month six, the day someone uploads a 4 GB MP4 over hotel Wi-Fi and your API container OOMs and takes the rest of the product with it.

This article is about the foundation pattern every serious file-handling system is built on: the client uploads directly to object storage, the API never touches the file bytes. We'll cover why, the contract that makes it safe, and the operational details that make it actually work in production.

The examples reference S3, but everything translates to GCS, Azure Blob, R2, and Backblaze B2 — they all expose the same signed-URL primitive.

The naive pattern, and why it dies ​

            ┌─────────┐  POST /upload   ┌─────────┐   PUT    ┌────┐
   Browser ─┤ Form    ├────────────────►│   API   ├─────────►│ S3 │
            │ (file)  │  multipart      │ (Go/Py/ │          └────┘
            └─────────┘                 │  Node)  │
                                        └─────────┘

The user POSTs a multipart form to your API. The API buffers (or streams) the body, possibly validates it, then writes to S3. Looks fine. Here's what breaks at scale:

  • Bandwidth doubles. Every uploaded byte travels client → API, then API → S3. A 1 GB file with 1,000 concurrent uploaders is 2 TB through your infrastructure instead of 1.
  • Memory pressure. Frameworks like Express and stdlib multipart.Reader will happily buffer a 5 GB part in RAM if you don't carefully stream. One careless line and your container's OOM-killed.
  • Tail latencies become your latencies. Your /upload endpoint's p99 is dominated by the slowest user's connection. The user on hotel Wi-Fi holds your API thread for 8 minutes.
  • No resumability. Connection drops at 90% → restart from zero. Frustrating for the user, expensive in support tickets.
  • Single-region bottleneck. Your API runs in us-east-1, your user is in Singapore. They upload over a 200 ms round trip even though S3 has a Singapore endpoint two hops away.
  • Hard to scale horizontally. Upload bandwidth becomes a coordination problem — load balancer sticky sessions, instance-level rate limits, etc.

You can paper over each problem (streaming parsers, larger instances, multi-region API, resumable upload middleware). At some point you realize you're rebuilding S3 in front of S3.

The pattern: direct-to-storage with signed URLs ​

The fix is to make the object store the upload endpoint, and reduce your API to a permission-granting service.

            ┌─────────┐  1. POST /upload-url       ┌─────────┐
   Browser ─┤ Request ├───────────────────────────►│   API   │
            │ slot    │                            │         │
            └─────────┘                            │ (auth,  │
                 ▲                                 │  quota, │
                 │ 2. signed URL +                 │  policy)│
                 │    DB record id                 └────┬────┘
                 │                                      │
                 │                            3. record │ "pending"
                 │                                      ▼
                 │                                 ┌─────────┐
                 │                                 │   DB    │
                 │                                 └─────────┘
                 ▼
            ┌─────────┐  4. PUT (file body)        ┌────────┐
            │ Direct  ├───────────────────────────►│   S3   │
            │ upload  │                            │ bucket │
            └─────────┘                            └───┬────┘
                                                       │
                                            5. event   ▼
                                                  ┌─────────┐
                                                  │ Worker  │ scan, thumbnail,
                                                  │         │ extract metadata,
                                                  └────┬────┘ mark "ready"
                                                       │
                                                       ▼
                                                  ┌─────────┐
                                                  │   DB    │ status=ready
                                                  └─────────┘

Six things happen, in order:

  1. Client asks for an upload slot. Sends metadata only: filename, declared size, declared content type, intent (avatar? KYC document? offer attachment?).
  2. API authenticates, authorizes, and validates. Is this user allowed to upload? Have they hit their quota? Is the declared size below the policy limit for this upload kind? Does the content type match the allow-list for the intent?
  3. API returns a signed URL. Plus a record id (so the client knows what to reference later) and any policy headers that S3 will enforce.
  4. Client uploads bytes directly to S3. Your API is out of the loop.
  5. S3 emits an event on completion (S3 Event Notifications → SNS/SQS/EventBridge, or in GCS-land Pub/Sub).
  6. A worker processes the object: virus scan, magic-byte content-type verification, thumbnail/transcode, metadata extraction, then marks the DB record ready.

Until step 6, the file exists but is "pending." The API never serves a download URL for a pending file. This separation is what makes the pattern safe.

Why this is the enterprise default ​

The bullet points sound abstract until you put numbers on them:

ConcernNaive (proxy)Direct-to-S3
API egress per 1 GB upload2 GB~1 KB (signed URL)
API memory per uploadtens of MB to GBconstant
ResumabilityDIYnative (S3 multipart)
Geographic optimizationone API regionclient → nearest S3 PoP (with Transfer Acceleration / multi-region buckets)
API instance size neededscales with upload concurrencytiny — handles metadata only
Time-to-first-byte for downloadAPI streamsclient gets a signed URL, fetches direct from CDN
Failure isolationbad upload kills requestbad upload is between client and S3, API unaffected

The cost story alone usually justifies the migration. AWS egress is $0.09/GB; if your API is in a different AZ from your bucket, you're also paying for that hop. Direct-to-S3 cuts that to zero.

Beyond cost, the architectural property that matters is separation of concerns. Your API is a permission gate. Object storage is the bulk-bytes layer. Workers are the post-processing layer. Each scales independently, each can be reasoned about and tested independently.

The signed-URL contract ​

Two flavors of signed upload exist in S3-land. Both work. Choose deliberately.

Presigned PUT ​

Server signs a single URL. Client PUTs the file body to it.

  • Simple to implement. One request, one URL.
  • Limited server-side enforcement: you can pin the exact object key and HTTP method; you can't easily constrain content-type or max size without additional headers the client must reproduce exactly.
  • Good for: single-shot uploads where you trust the client to behave (e.g., your own SPA, your own mobile app).

Presigned POST ​

Server signs a policy document describing what the client may upload. The client submits a multipart POST form whose fields match the policy.

  • Policy enforces constraints S3-side. Max size, content-type prefix, key prefix, required form fields — all enforced by S3 before any bytes are accepted. If the client lies about size or type, S3 rejects with 403 before the upload even starts.
  • More complex to consume on the client.
  • Good for: untrusted clients, public-facing upload widgets, anywhere policy enforcement at the storage layer matters more than client simplicity.

For enterprise use cases, presigned POST is the right default because the storage layer becomes your enforcement boundary — not your client code, which can be tampered with or reverse-engineered.

Large files: multipart with per-part signed URLs ​

Anything over ~100 MB should go through S3 multipart upload. The shape:

  1. Client asks API to initiate an upload — API calls CreateMultipartUpload, returns the UploadId and a per-part signing endpoint.
  2. Client splits the file into 5–100 MB parts. For each part, asks API for a signed UploadPart URL.
  3. Client uploads parts in parallel. On failure, just retry that part.
  4. When all parts are uploaded, client asks API to complete the upload. API calls CompleteMultipartUpload.
  5. API may also call abort on a timer for sessions that never complete (S3 has lifecycle rules for this too — set a "delete incomplete multipart uploads after 7 days" rule and forget about it).

This gives you free resumability (parallelize and retry parts), free parallelism (upload 4 parts simultaneously over 4 TCP connections), and a clean abort path.

Libraries that wrap this well: Uppy on the web, AWS SDKs' high-level transfer managers on mobile/desktop.

CORS and the origin model ​

Direct uploads require browser cross-origin requests to S3 / GCS. The setup that works (and the gotcha that bites everyone):

xml
<!-- S3 bucket CORS config -->
<CORSRule>
  <AllowedOrigin>https://app.your-domain.com</AllowedOrigin>
  <AllowedMethod>PUT</AllowedMethod>
  <AllowedMethod>POST</AllowedMethod>
  <AllowedHeader>*</AllowedHeader>
  <ExposeHeader>ETag</ExposeHeader>
  <ExposeHeader>x-amz-version-id</ExposeHeader>
  <MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>

ETag must be in ExposeHeader — multipart upload completion requires the client to send each part's ETag back to your API, and the browser strips response headers not listed in Access-Control-Expose-Headers. Without this, multipart silently fails at the complete step. This is the single most common direct-upload bug in production.

If you're using credentials: 'include' for the API (to send session cookies), your fetch to S3 should not include credentials — S3 doesn't accept them and the preflight will fail. The upload is authenticated by the signed URL, not by cookies.

Server-side guardrails: what to check before signing ​

The client tells you what it intends to upload. Trust nothing it says, but extract it cheaply.

Before issuing a signed URL:

  • AuthN/AuthZ. Standard request authentication, plus a domain check: can this user upload to this object scope? Per-tenant prefix, per-user prefix, per-object-type policy.
  • Quotas. Per-user, per-tenant, per-object-type — cheaper to refuse a signed URL than to receive 10 GB and refund the bytes after the fact.
  • Declared size below policy limit. If user says 5 GB and your policy is 100 MB for this upload kind, reject. (You don't trust the declared size for actual enforcement — that's S3's job via the POST policy — but rejecting early saves a round trip.)
  • Declared content type in the allow-list for the upload kind. Avatar uploads accept image/* only; KYC docs accept application/pdf and image/jpeg; etc.
  • Filename sanitization. Never use the user's filename as the storage key. Use a UUID or content-hash; keep the original filename in the metadata column.

At the storage layer (POST policy):

  • content-length-range: [0, maxBytes] — S3 enforces.
  • content-type-prefix: "image/" — S3 enforces.
  • x-amz-meta-* fields fixed — so the client can't tamper with metadata that downstream consumers trust.
  • Object key prefix fixed — client cannot upload to a different user's folder even if they try.

Post-upload (asynchronous, in the worker):

  • Magic-byte content-type verification. The declared image/png becomes ground truth only after libmagic confirms the actual bytes are PNG. People upload renamed executables.
  • Virus / malware scan. ClamAV in a Lambda, GCS via Google Cloud Storage Malware Scanner, or commercial solutions like Cloud Storage Security. Don't skip this for any user-uploaded content that another user might download.
  • Metadata extraction. Dimensions for images, duration for video, page count for PDFs.
  • Variant generation. Thumbnails, transcoded formats — the small/medium size variants we already do in this codebase.
  • Mark ready only after all of the above succeed. A failed scan transitions the record to quarantined, not deleted — keep it for audit.

Things to never accept from the client:

  • The final object key (lets attackers overwrite other users' files).
  • The "I'm done" signal as proof of completion (a malicious client can claim done without uploading; let S3's event tell you).
  • The reported file size, content type, or content hash as the ground truth (verify post-upload).

Encryption, retention, and lifecycle ​

Three knobs to turn before launch, not after the first compliance audit:

  • Encryption at rest. SSE-S3 is free and on by default in new buckets. For regulated data (PII, PCI, HIPAA-adjacent), use SSE-KMS with a customer-managed key — gives you key rotation, audit logging on key usage, and the ability to "delete" encrypted data instantly by destroying the key.
  • Versioning. Turn it on. Combined with object lock and lifecycle rules, this is your defense against both accidental deletion and ransomware that targets your storage layer.
  • Lifecycle rules. Automatic transitions: Standard → Standard-IA at 30d → Glacier Instant at 90d → Glacier Deep Archive at 1y. For most user-uploaded content the access pattern is "hot for a week, then forgotten" — lifecycle rules cut storage cost by 70-90% with zero application code.
  • Abort incomplete multipart uploads at 7 days. A one-line rule that prevents abandoned uploads from costing you forever.

Observability and audit ​

Things you'll want when (not if) something goes wrong:

  • Log every signed URL issuance. Who, what scope, what constraints, valid until when, what request id. When a leaked URL shows up in a referer log, you can trace it.
  • CloudTrail / GCS Audit Logs on the bucket are not optional. They are your tamper-evident record of every operation. Ship them somewhere out of reach of the same IAM principal that controls the bucket.
  • Metrics: signed-URL issuance rate, upload completion rate (started but never completed = abandoned), scan-failure rate, p50/p99 upload latency from the client's perspective (collected in the SPA, not the API).
  • Alarms: sudden spike in abandoned uploads (signed URLs not redeemed within 15 minutes), sudden spike in scan failures, presence of any object outside the allowed key prefixes (e.g., a daily Athena/BigQuery audit query).

When the API still has to touch bytes ​

The pattern has limits. The API gets back into the path when:

  • Server-side transformation is mandatory before storage. E.g., you must strip EXIF data from images before they're persisted. (Better: store both versions and serve the stripped one. Worker, not synchronous API.)
  • Authoritative size limits below S3's minimum. S3 doesn't enforce a minimum part size in single-part uploads, but if your business rule is "no files over 10 KB," that's so small that proxying is fine and simpler.
  • The upload must succeed-or-fail atomically with a DB transaction. Rare, and usually solved by treating the upload as a two-phase commit with compensation on the worker side rather than synchronously.

For everything else, direct-to-storage is the right default.

Tradeoffs and gotchas (honest list) ​

  • Debugging is split. The upload happens on a code path your server never sees. Reproducing client-side issues means correlating signed-URL issuance with S3 access logs by request id. Plan for this from day one — embed your request id in the object metadata via the policy.
  • You can no longer enforce business rules mid-stream. Once the upload is in flight, you can't change your mind. Validate before signing, validate again post-event. Don't try to validate during.
  • Clock skew breaks signatures. The 15-minute signature expiry assumes client and server clocks are sane. Mobile clients with bad clocks will fail in confusing ways — return a server timestamp in the slot response so the client can adjust.
  • CORS preflights count toward S3 request costs. Negligible for most workloads, occasionally surprising.
  • Per-part minimum size for multipart is 5 MB except for the last part. Splitting a 6 MB file into 1 MB parts will fail at completion time. Either enforce 5 MB+ part size, or fall back to single-shot for small files.
  • Signed URLs leak via referers and analytics. Treat them as bearer tokens — short expiry (5-15 min for uploads), HTTPS only, never log them, never embed in logs sent to third-party services.

What this looks like for our stack ​

In the i-filer-api codebase, the pattern is already in place:

  • POST /v1.0/auth/files/signurl → creates a DB record with status pending and returns a GCS signed URL. The client uploads directly to GCS; our API never sees the bytes.
  • Cloud Storage emits an event → Pub/Sub → the resize worker picks up the original, generates small and medium variants under the same ltree path, and marks the record ready.
  • Downloads use the same primitive in reverse: GET /v1.0/auth/files/:id?size=small issues a 60-second download signed URL, redirects, and the client streams directly from GCS.

The DB row is the index; GCS holds the bytes. The API mediates.

The remaining articles in this series will dig into the pieces this article skipped:

  1. Direct-to-S3 uploads (this article) — the foundation.
  2. Pre-flight validation, virus scanning, and the worker pipeline — how pending becomes ready safely.
  3. Multipart and resumable uploads in the browser — Uppy, tus, and the edge cases that bite.
  4. Variant generation: thumbnails, transcoding, OCR — the worker layer in depth, including the cost model.
  5. Signed download URLs, CDN integration, and access control — serving files at scale without your API in the path.
  6. Lifecycle, retention, and deletion — GDPR-compliant deletion when the bytes are encrypted with a per-tenant KMS key.
  7. Observability and forensics — what to log, how to find a leaked URL, how to prove a file was scanned.

TL;DR ​

If your file-upload endpoint is a multipart POST handler in your API, you are paying for it twice — in egress and in engineering effort papering over the consequences. Move the bytes off the API. Sign permission, let the object store accept the upload, let a worker decide whether to mark it ready. The architecture is the same whether you're at 100 uploads a day or 100,000 — only the worker concurrency changes.

References ​