Design a File Sync & Storage Service (Dropbox)

The whole difficulty is in NOT uploading files

File sync looks like "upload files to the cloud," but the entire difficulty is in not uploading files. A 2GB video where a user changes one tag should cost a few kilobytes of network, not 2GB. Two people uploading the same popular PDF should cost one copy of storage. Editing offline on a laptop and a phone must reconcile without silently losing an edit. The core techniques are chunking, dedup, delta sync, and conflict resolution.

Content-defined chunking

Instead of splitting a file into fixed 4MB blocks, CDC uses a rolling hash (Rabin fingerprint) over a sliding window and cuts a chunk boundary wherever the hash matches a pattern, yielding variable-size chunks averaging, say, 4MB. Why variable? Because if you insert one byte near the front of a file, fixed-size blocks all shift and every block hash changes, so the whole file re-uploads. CDC boundaries are anchored to content, so inserting a byte only changes the one chunk containing it; every other chunk keeps its old hash. Each chunk is hashed (SHA-256); the hash is both its content-address and its dedup key.

Dedup and delta sync

Store each unique chunk hash exactly once in the object store. A file becomes a manifest: an ordered list of chunk hashes. If two files (or two users, with global dedup) share chunks, they share storage. Delta sync falls straight out: to sync a changed file the client computes the new manifest, sends only the hashes to the server, the server replies which hashes it already has, and the client uploads only the missing chunks.

file --CDC--> [c1][c2][c3][c4]   each chunk -> SHA-256 -> content address
manifest = [h1, h2, h3, h4]
edit near start -> only c1 changes -> new manifest [h1', h2, h3, h4] -> upload 1 chunk

Metadata service and conflict resolution

Separate from blob storage, a metadata DB tracks: the file tree (paths, folders), each file's current manifest (chunk list) and version, per-device sync cursors, and sharing/ACLs. This is the coordination brain and needs strong consistency (a client must never see a manifest pointing at chunks that are not yet uploaded). The usual ordering: upload chunks to the object store first, then commit the metadata that references them.

Each file has a version vector or a monotonically increasing version. When a client uploads based on version N but the server is already at N+1 (someone else edited), that is a conflict. Dropbox's pragmatic answer is not to merge binary files: it keeps both, creating a "conflicted copy," so no edit is lost. The safe default is keep-both plus full version history so nothing is destroyed.

The client sync protocol

A local filesystem watcher detects changes, an upload queue chunks and pushes, a download queue applies remote changes, and a persisted cursor tracks the last-seen server state so an interrupted sync resumes instead of rescanning everything. Offline edits queue locally and reconcile on reconnect against the server version.

Interview nuance: the tempting-but-wrong move is to compute deltas on the server. You cannot, because the server does not have the client's new bytes until they are uploaded. The client computes the manifest and asks the server which chunks are missing (a "have/need" negotiation), so the expensive comparison happens before any bulk transfer.

Recap: content-defined chunking plus per-chunk hashing gives dedup and delta sync (upload only changed chunks), a strongly consistent metadata service maps files to chunk manifests and versions, and conflicts are resolved by keeping both copies plus history rather than merging blindly.