Title: Optimizing Vercel Sandbox snapshots - Vercel | BestBlogs.dev
URL Source: https://www.bestblogs.dev/article/45447ae1
Published Time: 2026-04-02 14:02:03
Markdown Content: Skip to main content Toggle navigation menu Toggle navigation menuArticlesPodcastsVideosTweetsSourcesNewsletters
⌘K
Change language Switch ThemeSign In
Narrow Mode
Optimizing Vercel Sandbox snapshots - Vercel
V Vercel News @Tom Lienardet al.
One Sentence Summary
Vercel optimized its Sandbox filesystem snapshots by implementing parallel S3 downloads, concurrent decompression, and local NVMe caching, reducing p75 restore latency from 40 seconds to under 1 second.
Summary
This article details the engineering journey of optimizing Vercel Sandbox snapshot restores. Initially prioritizing reliability, the team faced significant performance bottlenecks with p75 restore times exceeding 40 seconds. They achieved sub-second performance through a multi-pronged approach: parallelizing S3 downloads using Range headers, fanning out decompression across multiple Go goroutines, and streaming data directly from S3 to the decoder to eliminate intermediate disk I/O. Furthermore, by implementing a local NVMe disk cache with an LRU eviction policy to store decompressed images, they achieved a 95% cache hit rate, effectively bypassing the network and decompression overhead for most sandbox boots.
Main Points
* 1. Parallelizing the restore pipeline using S3 Range headers.By utilizing the HTTP Range header and the AWS Go SDK's transfermanager, the team downloaded snapshot chunks in parallel, resulting in 2-5x faster download speeds compared to sequential requests. * 2. Concurrent decompression and streaming data processing.Vercel moved from single-threaded to multi-goroutine decompression and piped S3 streams directly into the decoder, avoiding the latency of writing intermediate files to disk. * 3. Implementing a local NVMe disk cache for decompressed images.By caching the raw .img files on local fast storage with an LRU eviction policy, they achieved a 95% hit rate, allowing most restores to skip the download and decompression phases entirely.
Metadata
AI Score
87
Website vercel.com
Published At Today
Length 779 words (about 4 min)
Sign in to use highlight and note-taking features for a better reading experience. Sign in now
4 min read
Apr 2, 2026
Recently, we shipped filesystem snapshots in Vercel Sandbox, letting you capture and restore a sandbox's entire filesystem state. Snapshots need to be reliable, easy to use, and fast. Initially we focused on reliability, making sure we never fail to snapshot, restore, or lose data. But no one will use a stable product that is also dead slow. p75 snapshot restores were taking over 40 seconds. Now through parallelization and local caching, we got that under 1 second.
Link to headingWhat a snapshot looks like on disk
Vercel Sandbox runs on the same infrastructure as our internal builds product, Hive. Each sandbox is an isolated container inside a Firecracker microVM.
A snapshot is a compressed copy of the sandbox's disk. We're working with two different files:
* The raw disk image (.img), which can be several GBs
* A compressed version in our custom VHS format (Vercel Hive Snapshot), which is what gets uploaded to and downloaded from S3
When you call sandbox.snapshot(), we compress the .img into a .vhs and upload it to S3. When you call Sandbox.create() with a snapshot, we download the .vhs and decompress it back. The compression cuts the file size, which matters when you're moving hundreds of MBs to GBs over the network.
Link to headingParallelize you shall
With reliability in place, we turned to performance. The restore path was painfully sequential. We'd download the entire .vhs file from S3 in a single request, wait for it to finish, then decompress it in a single thread.
!Image 2: The original restore pipeline: a single S3 download followed by single-threaded decompression!Image 3: The original restore pipeline: a single S3 download followed by single-threaded decompression!Image 4: The original restore pipeline: a single S3 download followed by single-threaded decompression!Image 5: The original restore pipeline: a single S3 download followed by single-threaded decompression
The original restore pipeline: a single S3 download followed by single-threaded decompression
Snapshots range from 200MB to a few GBs, so that single S3 download alone could take several seconds to tens of seconds. We used the Range HTTP header to download chunks in parallel instead. The AWS Go SDK has a transfermanager API built for exactly this. After benchmarking various concurrency and chunk sizes, we ended up with 2-5x faster downloads.
!Image 6: Splitting the download into parallel S3 range requests!Image 7: Splitting the download into parallel S3 range requests!Image 8: Splitting the download into parallel S3 range requests!Image 9: Splitting the download into parallel S3 range requests
Splitting the download into parallel S3 range requests
Next up was decompression. Our .vhs format is composed of a header and a frame for each allocated region of the disk image. Instead of decoding and decompressing frames one by one, we switched to one decoder and N decompression goroutines. This made the .vhs to .img restore 2-4x faster, depending on snapshot size.
!Image 10: Fanning out decompression across multiple goroutines!Image 11: Fanning out decompression across multiple goroutines!Image 12: Fanning out decompression across multiple goroutines!Image 13: Fanning out decompression across multiple goroutines
Fanning out decompression across multiple goroutines
With both downloading and decompressing parallelized, we still had one remaining optimization. We piped S3 range request streams directly into decompression, without writing an intermediary file to disk or waiting for the full download to complete. That cut end-to-end restore time by another 2x.
!Image 14: Piping S3 download streams directly into decompression, no intermediate file!Image 15: Piping S3 download streams directly into decompression, no intermediate file!Image 16: Piping S3 download streams directly into decompression, no intermediate file!Image 17: Piping S3 download streams directly into decompression, no intermediate file
Piping S3 download streams directly into decompression, no intermediate file
Link to headingWe… didn't cache?
As you might have noticed, we so far only talked about improving the slow path, when we need to retrieve a snapshot from S3 on a cache miss. Well, we actually didn't have a fast path, so it was all cache misses. Yeah, we really didn't focus on performance at first.
Our sandboxes run on metal instances with NVMe disks, which means several terabytes of fast local storage that was mostly unused.
We added a local disk cache using LRU (least recently used) eviction, sized by total disk space rather than number of entries. We cache the decompressed .img directly rather than the compressed .vhs, so a cache hit skips both the download and the decompression. Once the cache fills up, the least recently used snapshots get evicted to make room.
Most customers have a "base" snapshot that they reuse across many sandboxes, so we're seeing a 95% cache hit rate. On a cache hit, boot time is bounded only by starting the microVM and container.
!Image 18: Local NVMe cache hit rate, consistently above 90%!Image 19: Local NVMe cache hit rate, consistently above 90%!Image 20: Local NVMe cache hit rate, consistently above 90%!Image 21: Local NVMe cache hit rate, consistently above 90%
Local NVMe cache hit rate, consistently above 90%
Link to headingFrom 40 seconds to sub-second
p75 dropped from 40s to sub-second, and p95 went from 50s to 5s. With our cache hit rate, most sandbox boots skip the download and decompression pipeline entirely.
!Image 22: Snapshot restore p95 latency dropping from 50s to under 10s!Image 23: Snapshot restore p95 latency dropping from 50s to under 10s!Image 24: Snapshot restore p95 latency dropping from 50s to under 10s!Image 25: Snapshot restore p95 latency dropping from 50s to under 10s
Snapshot restore p95 latency dropping from 50s to under 10s
We're exploring more ideas. Cache affinity would route sandboxes to metal instances that already have the requested snapshot cached, potentially eliminating the cold path for popular snapshots. But this risks thundering herds and hotspotting certain machines, so we're being deliberate about it.
Long term, we want the cold path fast enough that caching is a bonus, not a requirement.
These optimizations already power Automatic Persistence, now in beta. When you stop a named sandbox, its filesystem is automatically snapshotted and restored on resume. Sub-second restore means that cycle feels instant.
Filesystem snapshots are available today for all Vercel Sandboxes. Check the Sandbox documentation to get started.
V Vercel News @Tom Lienardet al.
One Sentence Summary
Vercel optimized its Sandbox filesystem snapshots by implementing parallel S3 downloads, concurrent decompression, and local NVMe caching, reducing p75 restore latency from 40 seconds to under 1 second.
Summary
This article details the engineering journey of optimizing Vercel Sandbox snapshot restores. Initially prioritizing reliability, the team faced significant performance bottlenecks with p75 restore times exceeding 40 seconds. They achieved sub-second performance through a multi-pronged approach: parallelizing S3 downloads using Range headers, fanning out decompression across multiple Go goroutines, and streaming data directly from S3 to the decoder to eliminate intermediate disk I/O. Furthermore, by implementing a local NVMe disk cache with an LRU eviction policy to store decompressed images, they achieved a 95% cache hit rate, effectively bypassing the network and decompression overhead for most sandbox boots.
Main Points
* 1. Parallelizing the restore pipeline using S3 Range headers.
By utilizing the HTTP Range header and the AWS Go SDK's transfermanager, the team downloaded snapshot chunks in parallel, resulting in 2-5x faster download speeds compared to sequential requests.
* 2. Concurrent decompression and streaming data processing.
Vercel moved from single-threaded to multi-goroutine decompression and piped S3 streams directly into the decoder, avoiding the latency of writing intermediate files to disk.
* 3. Implementing a local NVMe disk cache for decompressed images.
By caching the raw .img files on local fast storage with an LRU eviction policy, they achieved a 95% hit rate, allowing most restores to skip the download and decompression phases entirely.
Key Quotes
* Initially we focused on reliability, making sure we never fail to snapshot, restore, or lose data. But no one will use a stable product that is also dead slow. * We piped S3 range request streams directly into decompression, without writing an intermediary file to disk or waiting for the full download to complete. * We added a local disk cache using LRU (least recently used) eviction... caching the decompressed .img directly rather than the compressed .vhs.
AI Score
87
Website vercel.com
Published At Today
Length 779 words (about 4 min)
Tags
Vercel
Performance Optimization
Firecracker
S3
Go
Related Articles
* Fast regex search: indexing text for agent tools * Security boundaries in agentic architectures - Vercel * “Anyone can cook”: How v0 is bringing git workflows to vibe-coding | Guillermo Rauch (Vercel CEO) * The Shift to Agentic Engineering: Insights from OpenClaw Creator * Friend Bubbles: Enhancing Social Discovery on Facebook Reels * AGENTS.md outperforms skills in our agent evals - Vercel * Build knowledge agents without embeddings - Vercel * How we run Vercel's CDN in front of Discourse - Vercel * How we made v0 an effective coding agent - Vercel * How we rebuilt Next.js with AI in one week HomeArticlesPodcastsVideosTweets