Resumable Cloud-Storage Sync Engine

Why the first version stops working

Syncing a storage account looks like a recursive walk plus a download loop. It is — right up until the account is large enough, and then each of these bites in turn. None are hypothetical; each is a different incident with a different signature.

Recursion dies at the platform timeout, and restarts at the root. A recursive walk over a big tree gets killed mid-flight by a serverless maxDuration. The next invocation begins again at the account root and re-scans the same early slice, forever. The tree is never finished, and the symptom is "sync seems slow" rather than "sync is broken" — which is why it survives so long. The fix is an explicit frame stack you can persist. Resuming continues mid-folder, not just mid-tree: restarting a 40-page folder on every run is the same bug in miniature.

A dead file is retried forever. A 404 and a 503 are not the same failure. Treat both as "retry later" and a handful of permanently-deleted files consume every run indefinitely, with no chance of success. Failures are classified — permanent (stop), transient (backoff), throttled (back off harder and longer), unknown (retry twice, then stop, because an unrecognised error that recurs is almost always permanent).

An orphaned claim blocks an item forever — and the obvious clock makes it worse. Marking an item in-flight prevents double-processing, but a worker that dies leaves the claim set. Recovering by "claimed more than N minutes ago" fails if you reuse the discovery timestamp, because re-walking the tree refreshes it and the orphan looks permanently fresh. The claim gets its own clock.

Dedupe by content, never by name. `lease.pdf` in twelve folders is twelve different leases. Collapsing by name loses eleven of them and there is no way to notice afterwards. Only identical bytes are a duplicate, and a file with no hash is always kept — unknown is not the same as duplicate.

A big file can OOM the whole run. Extraction libraries routinely allocate many times a file's size in memory; one 80MB PDF takes down the process and every other item in the batch with it. The size check happens *before* bytes are loaded, and the format list is an allowlist — a denylist lets every unseen format into the pipeline to fail forever as a permanently-stuck item nobody notices.

Also

Progress is reported as counts, never a percentage: the denominator is unknown until the walk finishes, and a fabricated percentage that sticks at 90% is worse than a number that is true. Unreadable folders are recorded rather than skipped silently, so a partial sync can never be mistaken for a complete one.

Verified by mutation

Every rule had its guard removed one at a time and the suite was confirmed to fail — 9 of 9, including the two that cost the most: the cursor that stops resuming, and dedupe collapsing by filename.

31 tests passing · typecheck clean · 10/10 mutations caught.

Delivery

Private repository invite within 24 hours. Single-product commercial license: use and modify in one shipped product; no redistribution or resale of the source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function startCursor(rootId: string): WalkCursor;
  export async function walkTree( port: StoragePort, cursor: WalkCursor, opts: WalkOptions = {}, ): Promise<WalkResult>;
  export function classifyFailure(err: unknown): FailureClass;
  export function nextAttempt( failure: FailureClass, attempts: number, policy: Record<FailureClass, RetryPolicy> = DEFAULT_RETRY, ):;
  export function tryClaim( existing: ClaimRecord | null, workerId: string, nowMs: number, ttlMs: number = DEFAULT_CLAIM_TTL_MS, ): ClaimDecision;
  export function dedupeByContent(nodes: TreeNode[]): DedupeResult;
  export function shouldExtract(node: TreeNode, limits: ExtractionLimits = DEFAULT_LIMITS): ExtractDecision;
  export function progressOf(cursor: WalkCursor): SyncProgress;

  export type FailureClass = "permanent" | "transient" | "throttled" | "unknown";