What the service actually has to guarantee
Start from obligations rather than components. The service must accept files ranging from a few kilobytes to multiple terabytes; complete an upload across an unreliable network without restarting from zero; never hand back a success for a file it stored incorrectly; make a stored file retrievable by an identifier that does not change; enforce who may write what and how much; and eventually delete things, correctly, when asked or when policy says so.
Two quantities shape everything else. The size distribution is almost always bimodal -- a very large count of small files and a small count of very large ones -- and the two ends want opposite optimisations: small files want one round trip and low per-object overhead, large files want chunking and parallelism. Design for both explicitly with a size threshold, rather than forcing one path to serve both. The write-to-read ratio decides how much you invest in the read path; a media service reads each object thousands of times and needs a CDN, a backup service may never read most objects at all and should optimise for cheap cold storage.
Non-negotiable properties worth stating up front: an upload that reports success is durable and complete; the same upload attempted twice does not produce two objects or two charges; and a partially completed upload is either resumable or garbage-collected, never a permanent invisible cost.
Control plane and data plane must be separate
The single most important structural decision is that your application servers do not carry the bytes. They authenticate the request, apply policy, record metadata, and hand the client a credential that lets it talk to the storage system directly. Bytes flow client-to-storage; only decisions flow through you.
The mechanism is the pre-signed URL: a storage URL carrying a signature that encodes the permitted operation, the exact object key, an expiry, and optionally conditions such as a content-length range or a required content type. The client uses it once and it expires. Nothing about your service is in the transfer path, so a thousand concurrent hundred-gigabyte uploads consume no application capacity at all.
Proxying instead is a decision with consequences that arrive later. Your servers become bandwidth-bound and must be scaled on transfer volume rather than request rate; a slow client occupies a connection and a buffer for minutes; a deploy interrupts in-flight uploads. There are legitimate reasons to proxy -- inline scanning or transformation before anything is persisted, or a compliance requirement that no client ever holds a storage credential -- and if you have one, isolate the proxying tier from the rest of the API so its scaling profile does not contaminate everything else.
Scope pre-signed credentials narrowly. Sign for a specific key rather than a prefix, keep expiry in minutes and not days, pin a content-length range so a client cannot upload a terabyte into a slot meant for an avatar, and never sign an operation the caller was not authorised for at issue time -- the signature is checked by storage, which knows nothing about your permission model.
The multipart protocol
For anything above a threshold -- commonly somewhere between 8 and 100 megabytes -- the upload is split. The three-call shape is standard: InitiateMultipartUpload returns an upload ID; UploadPart is called once per chunk with that ID and a part number, returning an entity tag per part; CompleteMultipartUpload submits the ordered list of part numbers and tags, and the storage system assembles them into one object.
The constraints in S3 are representative and worth knowing because they bound your chunk-size arithmetic: parts are at least 5 MiB except the last, at most 5 GiB each, at most 10,000 per upload, and the assembled object may reach 5 TiB. Ten thousand parts is the constraint that bites -- a fixed 5 MiB chunk size caps you at about 50 GiB, so chunk size has to be derived from the file size rather than hard-coded. A simple rule works: pick the larger of your minimum chunk and fileSize / 9000, rounded up to a convenient boundary.
Parts upload in parallel, which is where the throughput comes from, and each part retries independently, which is where the reliability comes from. Concurrency of four to eight is a reasonable default; more helps on fat long-distance links and hurts on mobile connections where it induces congestion and drains battery. Adaptive concurrency -- raise while throughput improves, back off on timeouts -- is a meaningful win for a client SDK.
Complete is the commit point and must be treated as one. Until it succeeds the object does not exist to readers; after it succeeds the object is whole. Make the call idempotent in your own layer by keying it on the upload ID, because a client that times out on complete and retries must not create a second object.
Resumability across days, not just retries
Part-level retry handles a blip. Resumability handles a laptop closing, a phone losing signal in a tunnel, or a user coming back tomorrow. The design requirement is that the authoritative record of what has been uploaded lives on the server, because the client's record is exactly the thing that gets lost.
The flow is: the client reconnects, presents the upload ID, asks which parts the service already holds, and uploads only the missing ones. Storage systems expose this directly -- listing the parts of an in-progress multipart upload -- and a resumable session URI in other APIs serves the same purpose by reporting the committed byte offset. Either way the client needs a durable local record of the upload ID and the chunk boundaries, written to disk before the first part goes out, so that a process restart can rejoin.
Chunk boundaries must be deterministic for resumption to work: part 7 must cover the same byte range tomorrow as it did today. Derive boundaries from the file size with a fixed rule, and detect the case where the underlying file changed between sessions by comparing size and modification time, or better, a checksum -- resuming into a file that was edited produces a corrupt object that passes every part-level check.
Give sessions an explicit lifetime, expose it, and expire them. A week is a common choice. This is where the industry's most common silent cost bug lives: abandoned multipart uploads occupy storage that is billed but invisible in an object listing. A lifecycle rule that aborts incomplete uploads after a fixed number of days is a one-line configuration that has saved organisations substantial monthly spend, and it should be in the initial deployment rather than added after the first audit.
Integrity — checking that what arrived is what was sent
Retries protect against connections dropping. They do not protect against a proxy corrupting a byte, a client bug truncating a buffer, or a bad memory module. Integrity requires an explicit checksum path, and it has to be end to end.
Check at two levels. Per part, the client computes a checksum before sending and includes it with the request; the storage system verifies on receipt and rejects a mismatch, so corruption is caught within one chunk instead of after a multi-hour transfer. Modern object stores support several algorithms for this and CRC32C is a good default -- it is hardware-accelerated and this is an accidental-corruption check, not an adversarial one. Per object, the client computes a checksum of the whole file and stores it as metadata, giving you a value to verify against on download and during periodic scrubbing.
One detail catches nearly everyone: the entity tag of a multipart object is not a hash of the object's contents. It is derived from the part checksums and carries a part count suffix, so comparing it to a locally computed digest of the file always fails. If your verification story is 'compare the ETag', it works in testing on small single-part uploads and breaks the moment a real file crosses the multipart threshold. Store your own full-object checksum in metadata and compare against that.
Deduplication and content addressing
Hash the content, store one copy per distinct hash, and let many logical files point at it. The metadata layer keeps a mapping from user-visible file identity to a content hash, and the storage layer keeps one blob per hash. Savings on typical user-generated content are large -- the same attachments, installers and shared media recur constantly -- and the mechanism also gives you a free integrity check and a natural cache key.
Granularity is the design choice. Whole-file dedup is trivial to implement and catches only exact duplicates. Fixed-size block dedup catches more, but a single byte inserted at the front shifts every subsequent block and defeats it. Content-defined chunking solves that: a rolling hash over a sliding window sets chunk boundaries wherever the hash matches a pattern, so an insertion changes only the chunks around it. This is what backup and sync systems use, and it is what makes uploading a slightly edited large file cheap. The cost is complexity and a chunk index that can grow larger than you expect.
Deletion is the part that is always underestimated. With sharing, a blob may not be deleted until the last reference goes away, which means reference counting -- and reference counts race with concurrent uploads that are about to add a reference to a blob you have just decided is unreferenced. The safe pattern is mark-and-sweep with a grace period: mark candidates, wait longer than the longest possible in-flight upload, re-verify, then delete. Immediate deletion on a count reaching zero is the design that eventually loses someone's file.
The security caveat in cross-user deduplication
Client-side dedup -- where the client hashes first and the server replies 'already have it, skip the upload' -- is the most attractive version, because the bytes never cross the network at all. It is also a documented information leak, and it is worth understanding before shipping it.
The leak is that upload speed becomes an oracle. An attacker who can construct a candidate file and observe that the server skipped the transfer learns that this exact file already exists in the system. Where the file has low entropy -- a form letter with a name and an amount filled in, a document with a small number of plausible variants -- the attacker can enumerate candidates and confirm the true one. A related attack claims ownership of a file knowing only its hash, which is not a secret, and thereby obtains content it never had.
The mitigations are known. Restrict dedup to within a security boundary -- per user, per tenant, per organisation -- which keeps most of the savings in the environments where duplicates actually cluster and removes the cross-tenant oracle entirely. Require proof of ownership: before honouring a dedup claim, challenge the client to produce hashes of randomly chosen byte ranges, which it can only do if it genuinely holds the file. Or perform dedup server-side only, after the upload has happened, which forfeits the bandwidth saving but keeps the entire storage saving and has no oracle at all. Server-side-only is the right default unless bandwidth is the binding constraint.
Metadata, and staying consistent with the object store
The metadata database is the system of record for identity and permissions; the object store is the system of record for bytes. They are two systems, they fail independently, and there is no transaction spanning them. Every upload service has to decide which inconsistency it prefers.
A workable schema has four concerns separated: uploads (session state, part progress, expiry), blobs (content hash, size, storage location, reference count), files (user-visible name, owner, parent folder, version, pointer to a blob), and events for the async pipeline. Separating file identity from blob identity is what makes dedup, versioning and server-side copy cheap -- a copy is a new file row pointing at the same blob.
Order the writes so the failure mode is a harmless orphan rather than a dangling pointer: write the bytes first, commit the metadata second. If the process dies in between, you have an unreferenced blob, which a sweeper reclaims. The reverse order leaves a metadata row promising an object that does not exist, which surfaces to users as a broken file. Run the orphan sweeper on a schedule, give it the same grace period as the deletion sweeper, and instrument it -- a sudden rise in orphans is an early signal that completions are failing.
Make completion idempotent with a client-supplied idempotency key stored alongside the upload session, returning the original result on a repeat. Retries at the commit boundary are the normal case, not the exception.