The naive upload service is one endpoint that accepts a request body and writes it somewhere. It works until a file is large enough that a dropped connection at ninety percent means starting over, until enough clients upload at once that your application servers are spending their memory and bandwidth being a very expensive pipe, and until someone asks why storage costs are growing faster than the number of distinct files. Every serious design answers those three pressures with the same three moves: get the bytes out of your application's path, break the transfer into independently retryable pieces, and address content by what it is rather than by who uploaded it. The rest of the design -- integrity, metadata consistency, lifecycle, the post-upload pipeline -- follows from those decisions.
What the service actually has to guarantee
Start from obligations rather than components. The service must accept files ranging from a few kilobytes to multiple terabytes; complete an upload across an unreliable network without restarting from zero; never hand back a success for a file it stored incorrectly; make a stored file retrievable by an identifier that does not change; enforce who may write what and how much; and eventually delete things, correctly, when asked or when policy says so.
Two quantities shape everything else. The size distribution is almost always bimodal -- a very large count of small files and a small count of very large ones -- and the two ends want opposite optimisations: small files want one round trip and low per-object overhead, large files want chunking and parallelism. Design for both explicitly with a size threshold, rather than forcing one path to serve both. The write-to-read ratio decides how much you invest in the read path; a media service reads each object thousands of times and needs a CDN, a backup service may never read most objects at all and should optimise for cheap cold storage.
Non-negotiable properties worth stating up front: an upload that reports success is durable and complete; the same upload attempted twice does not produce two objects or two charges; and a partially completed upload is either resumable or garbage-collected, never a permanent invisible cost.
Control plane and data plane must be separate
The single most important structural decision is that your application servers do not carry the bytes. They authenticate the request, apply policy, record metadata, and hand the client a credential that lets it talk to the storage system directly. Bytes flow client-to-storage; only decisions flow through you.
The mechanism is the pre-signed URL: a storage URL carrying a signature that encodes the permitted operation, the exact object key, an expiry, and optionally conditions such as a content-length range or a required content type. The client uses it once and it expires. Nothing about your service is in the transfer path, so a thousand concurrent hundred-gigabyte uploads consume no application capacity at all.
Proxying instead is a decision with consequences that arrive later. Your servers become bandwidth-bound and must be scaled on transfer volume rather than request rate; a slow client occupies a connection and a buffer for minutes; a deploy interrupts in-flight uploads. There are legitimate reasons to proxy -- inline scanning or transformation before anything is persisted, or a compliance requirement that no client ever holds a storage credential -- and if you have one, isolate the proxying tier from the rest of the API so its scaling profile does not contaminate everything else.
Scope pre-signed credentials narrowly. Sign for a specific key rather than a prefix, keep expiry in minutes and not days, pin a content-length range so a client cannot upload a terabyte into a slot meant for an avatar, and never sign an operation the caller was not authorised for at issue time -- the signature is checked by storage, which knows nothing about your permission model.
The multipart protocol
For anything above a threshold -- commonly somewhere between 8 and 100 megabytes -- the upload is split. The three-call shape is standard: InitiateMultipartUpload returns an upload ID; UploadPart is called once per chunk with that ID and a part number, returning an entity tag per part; CompleteMultipartUpload submits the ordered list of part numbers and tags, and the storage system assembles them into one object.
The constraints in S3 are representative and worth knowing because they bound your chunk-size arithmetic: parts are at least 5 MiB except the last, at most 5 GiB each, at most 10,000 per upload, and the assembled object may reach 5 TiB. Ten thousand parts is the constraint that bites -- a fixed 5 MiB chunk size caps you at about 50 GiB, so chunk size has to be derived from the file size rather than hard-coded. A simple rule works: pick the larger of your minimum chunk and fileSize / 9000, rounded up to a convenient boundary.
Parts upload in parallel, which is where the throughput comes from, and each part retries independently, which is where the reliability comes from. Concurrency of four to eight is a reasonable default; more helps on fat long-distance links and hurts on mobile connections where it induces congestion and drains battery. Adaptive concurrency -- raise while throughput improves, back off on timeouts -- is a meaningful win for a client SDK.
Complete is the commit point and must be treated as one. Until it succeeds the object does not exist to readers; after it succeeds the object is whole. Make the call idempotent in your own layer by keying it on the upload ID, because a client that times out on complete and retries must not create a second object.
Resumability across days, not just retries
Part-level retry handles a blip. Resumability handles a laptop closing, a phone losing signal in a tunnel, or a user coming back tomorrow. The design requirement is that the authoritative record of what has been uploaded lives on the server, because the client's record is exactly the thing that gets lost.
The flow is: the client reconnects, presents the upload ID, asks which parts the service already holds, and uploads only the missing ones. Storage systems expose this directly -- listing the parts of an in-progress multipart upload -- and a resumable session URI in other APIs serves the same purpose by reporting the committed byte offset. Either way the client needs a durable local record of the upload ID and the chunk boundaries, written to disk before the first part goes out, so that a process restart can rejoin.
Chunk boundaries must be deterministic for resumption to work: part 7 must cover the same byte range tomorrow as it did today. Derive boundaries from the file size with a fixed rule, and detect the case where the underlying file changed between sessions by comparing size and modification time, or better, a checksum -- resuming into a file that was edited produces a corrupt object that passes every part-level check.
Give sessions an explicit lifetime, expose it, and expire them. A week is a common choice. This is where the industry's most common silent cost bug lives: abandoned multipart uploads occupy storage that is billed but invisible in an object listing. A lifecycle rule that aborts incomplete uploads after a fixed number of days is a one-line configuration that has saved organisations substantial monthly spend, and it should be in the initial deployment rather than added after the first audit.
Integrity — checking that what arrived is what was sent
Retries protect against connections dropping. They do not protect against a proxy corrupting a byte, a client bug truncating a buffer, or a bad memory module. Integrity requires an explicit checksum path, and it has to be end to end.
Check at two levels. Per part, the client computes a checksum before sending and includes it with the request; the storage system verifies on receipt and rejects a mismatch, so corruption is caught within one chunk instead of after a multi-hour transfer. Modern object stores support several algorithms for this and CRC32C is a good default -- it is hardware-accelerated and this is an accidental-corruption check, not an adversarial one. Per object, the client computes a checksum of the whole file and stores it as metadata, giving you a value to verify against on download and during periodic scrubbing.
One detail catches nearly everyone: the entity tag of a multipart object is not a hash of the object's contents. It is derived from the part checksums and carries a part count suffix, so comparing it to a locally computed digest of the file always fails. If your verification story is 'compare the ETag', it works in testing on small single-part uploads and breaks the moment a real file crosses the multipart threshold. Store your own full-object checksum in metadata and compare against that.
Deduplication and content addressing
Hash the content, store one copy per distinct hash, and let many logical files point at it. The metadata layer keeps a mapping from user-visible file identity to a content hash, and the storage layer keeps one blob per hash. Savings on typical user-generated content are large -- the same attachments, installers and shared media recur constantly -- and the mechanism also gives you a free integrity check and a natural cache key.
Granularity is the design choice. Whole-file dedup is trivial to implement and catches only exact duplicates. Fixed-size block dedup catches more, but a single byte inserted at the front shifts every subsequent block and defeats it. Content-defined chunking solves that: a rolling hash over a sliding window sets chunk boundaries wherever the hash matches a pattern, so an insertion changes only the chunks around it. This is what backup and sync systems use, and it is what makes uploading a slightly edited large file cheap. The cost is complexity and a chunk index that can grow larger than you expect.
Deletion is the part that is always underestimated. With sharing, a blob may not be deleted until the last reference goes away, which means reference counting -- and reference counts race with concurrent uploads that are about to add a reference to a blob you have just decided is unreferenced. The safe pattern is mark-and-sweep with a grace period: mark candidates, wait longer than the longest possible in-flight upload, re-verify, then delete. Immediate deletion on a count reaching zero is the design that eventually loses someone's file.
The security caveat in cross-user deduplication
Client-side dedup -- where the client hashes first and the server replies 'already have it, skip the upload' -- is the most attractive version, because the bytes never cross the network at all. It is also a documented information leak, and it is worth understanding before shipping it.
The leak is that upload speed becomes an oracle. An attacker who can construct a candidate file and observe that the server skipped the transfer learns that this exact file already exists in the system. Where the file has low entropy -- a form letter with a name and an amount filled in, a document with a small number of plausible variants -- the attacker can enumerate candidates and confirm the true one. A related attack claims ownership of a file knowing only its hash, which is not a secret, and thereby obtains content it never had.
The mitigations are known. Restrict dedup to within a security boundary -- per user, per tenant, per organisation -- which keeps most of the savings in the environments where duplicates actually cluster and removes the cross-tenant oracle entirely. Require proof of ownership: before honouring a dedup claim, challenge the client to produce hashes of randomly chosen byte ranges, which it can only do if it genuinely holds the file. Or perform dedup server-side only, after the upload has happened, which forfeits the bandwidth saving but keeps the entire storage saving and has no oracle at all. Server-side-only is the right default unless bandwidth is the binding constraint.
Metadata, and staying consistent with the object store
The metadata database is the system of record for identity and permissions; the object store is the system of record for bytes. They are two systems, they fail independently, and there is no transaction spanning them. Every upload service has to decide which inconsistency it prefers.
A workable schema has four concerns separated: uploads (session state, part progress, expiry), blobs (content hash, size, storage location, reference count), files (user-visible name, owner, parent folder, version, pointer to a blob), and events for the async pipeline. Separating file identity from blob identity is what makes dedup, versioning and server-side copy cheap -- a copy is a new file row pointing at the same blob.
Order the writes so the failure mode is a harmless orphan rather than a dangling pointer: write the bytes first, commit the metadata second. If the process dies in between, you have an unreferenced blob, which a sweeper reclaims. The reverse order leaves a metadata row promising an object that does not exist, which surfaces to users as a broken file. Run the orphan sweeper on a schedule, give it the same grace period as the deletion sweeper, and instrument it -- a sudden rise in orphans is an early signal that completions are failing.
Make completion idempotent with a client-supplied idempotency key stored alongside the upload session, returning the original result on a repeat. Retries at the commit boundary are the normal case, not the exception.
The post-upload pipeline
Most of what a file upload service does happens after the upload. Virus scanning, thumbnail and preview generation, transcoding, text extraction and indexing, metadata enrichment, replication to a second region. None of it belongs in the request path, because it is slow, it fails, and it needs to be retried independently.
The standard shape is an event emitted on completion into a durable queue, consumed by workers per concern. Give the file an explicit state machine -- uploading, pending_scan, available, quarantined, failed -- and make the download path respect it. The subtle and important rule is that a file is not downloadable until scanning has cleared it, since serving an unscanned file to a second user is exactly how a storage service becomes a malware distribution channel.
Workers must be idempotent, because the queue will deliver twice. Key the output on content hash plus operation, and check whether the artefact already exists before doing the work -- this also gives you free dedup on derived artefacts, since two identical uploads share thumbnails as well as bytes. Bound the work per item and route repeated failures to a dead-letter queue with the file identifier attached; a single twelve-gigabyte video that no transcoder can handle should not stall the pipeline for everyone else.
Lifecycle, storage classes and the cost model
Storage cost is not one number. Objects can live in tiers whose price differs by an order of magnitude, with retrieval latency and per-request cost varying inversely. Access patterns for uploaded content are strongly time-decaying -- most objects are read heavily in their first days and rarely thereafter -- which is precisely the shape lifecycle transitions exist for.
A workable policy: keep objects in standard storage while access is likely, transition to an infrequent-access tier after a month, to archival after a quarter or a year, and expire or delete according to your retention obligation. Two cautions. Transitions and small objects interact badly -- per-object transition charges and per-tier minimum billable sizes can make moving millions of tiny files cost more than leaving them -- so apply transitions above a size threshold. And archival retrieval is slow and separately billed, so anything a user can request on demand should not be archived without a restore flow they can see.
Alongside transitions, three rules belong in every deployment: abort incomplete multipart uploads after a fixed number of days, expire noncurrent object versions if versioning is on, and expire delete markers. Versioning is worth enabling -- it makes accidental deletion recoverable and gives you a soft-delete window -- but without noncurrent-version expiry it means you never actually delete anything, which is a surprise that arrives as a bill rather than as an error.
Scale, hot spots and the read path
Object stores partition by key prefix and publish per-prefix request rate ceilings -- on S3, in the thousands of writes and several thousand reads per second per prefix, with the partitioning adapting over time. A key scheme that begins with a timestamp puts every write in the same prefix and hits that ceiling while the rest of the namespace idles. Leading the key with something high-cardinality -- a hash prefix, a tenant identifier, the content hash itself if you are content-addressing -- distributes the load. Content addressing gets this right for free, which is a underrated secondary benefit.
Downloads should not come from the origin. A CDN in front of the bucket converts read scaling into a cached problem and moves egress to a cheaper path; signed CDN URLs preserve access control. For large downloads support range requests so clients resume, and set cache headers deliberately -- content-addressed objects are immutable and can be cached effectively forever, which is another reason to expose the hash in the URL.
Two client-side realities deserve design attention because they generate most support load. Mobile uploads are interrupted constantly, so smaller chunks, aggressive session persistence and background-transfer APIs matter more than raw throughput. Very large single files from a browser hit memory limits if the implementation reads the file before slicing it; slice from the file handle and stream each part. Both are places where the protocol is correct and the experience is still bad unless the client is written with the same care as the server.