How Cloudflare Freed 100 TB of Memory by Changing One Cache Entry

One wasted byte sounds like nothing. Across 250 billion objects it is 250 gigabytes.

That multiplication is the whole reason this topic is worth your time. Cloudflare runs the 1.1.1.1 public resolver, and it says the cache behind it holds over 250 billion DNS entries at any moment. In August 2026 the team published what happened when they took one cache entry apart and rebuilt its representation. The entry went from 953 bytes to 420, a 56% reduction. Across the fleet, roughly 100 terabytes of memory stopped being used. Cloudflare puts that at about the RAM in 130 of its Gen 13 servers.

The interesting part is that none of the five changes was clever in the way engineers usually mean. There is no compression algorithm, no new data structure, no exotic library. Every change was a different answer to the same question: what is this object actually paying for, and does it still need to pay for it?

I want to work through that question properly, because it applies far beyond DNS. If you hold millions or billions of anything in memory, the same accounting applies. So this starts at bytes and pointers, builds up through container layout and allocators, goes through the five changes one at a time, then explains why smaller also turned out to be faster, and ends with how to run the same exercise on your own system.

A note on sourcing before anything else. Numbers attributed to Cloudflare come from its published engineering posts, linked at the end. Everything else is reasoning from those numbers or general systems knowledge, and I say which is which. This is a reading of someone else's excellent work, not a report of mine.

First, what the result actually says

Cloudflare's headline benchmark numbers:

Metric	Before	After	Change
Net footprint per entry	953 B	420 B	-56%
Allocated bytes per entry	1.1 KB	461 B	-58%
Cache insert throughput	625k/s	893k/s	+43%
Cache lookup latency	828 ns	670 ns	-19%

And in production, after the rollouts settled, instance memory at p99 fell from 9.3 GB to 5.3 GB, and at p90 from 6.5 GB to 3.8 GB.

Now the correction that most retellings of this get wrong. A lot of summaries turned "100 TB freed" into "Cloudflare saved millions of dollars." Cloudflare did not publish that, and it does not follow. What Cloudflare said is that it intends to put the freed memory back to work as larger caches, so hit rates improve and fewer queries go upstream.

Those are four different things and it is worth keeping them apart:

Concept	What it means	Can you read it from the post?
Freed working set	RAM the same workload no longer occupies	Yes, about 100 TB
Capacity equivalent	How much installed RAM that equals	Yes, about 130 servers' worth
Avoided purchase	Hardware you now never buy	No, depends on the plan
Cash saved	Actual money	Not published

The other thing worth saying early: this is not a Rust story. Rust made the representation visible and gave the team good tools for expressing owned immutable data without a garbage collector. But every underlying idea works in C, C++, Go, Java, C# or a database engine. Stop paying for mutability you no longer use. Do not store the same thing twice. Make the common case cheap and the rare case expensive. Keep hot data together. Decode once instead of every time.

Where memory actually goes

Most of us carry a mental model that looks like this:

object size = sum of the field sizes

That model is wrong in a way that does not matter at a thousand objects and matters enormously at a billion. The real accounting is closer to:

useful payload
+ struct padding for alignment
+ container metadata (pointer, length, capacity)
+ allocator metadata and size-class rounding
+ allocation capacity you reserved but never used
+ fragmentation
+ the same value stored more than once

Let me take those one at a time, because each one turns into a specific optimization later.

Padding

Hardware wants values to start at addresses that are a multiple of their size. Compilers insert padding to make that true, and round the whole struct up to its own alignment.

struct Example {
    a: u8,   // 1 byte
    b: u64,  // 8 bytes, wants an 8-byte boundary
    c: u8,   // 1 byte
}

The fields add up to 10 bytes. The struct is bigger, because b cannot start at offset 1, and the struct has to end on a boundary too. Reorder the fields so the small ones sit together and the same data gets smaller. That is free, and almost nobody checks it.

Container metadata

A growable array has to remember three things: where the buffer is, how many elements are live, and how many fit before it must reallocate. In Rust that is Vec<T>, and the shape is the same in every language that has one.

Vec<T> control value
+-------------------+
| ptr       8 bytes |----> [T][T][T][unused][unused]
| len       8 bytes |
| capacity  8 bytes |
+-------------------+

Twenty-four bytes of bookkeeping before a single element exists, plus whatever spare capacity the buffer is holding. A String is the same thing with a promise that the bytes are valid UTF-8, so it carries the same three words.

That capacity field is the interesting one. It exists to answer a question only a growable object asks: how much more can I take before I have to move? An object that will never grow again is paying eight bytes, forever, to answer a question nobody will ask it.

The allocator

When you ask for 37 bytes, you do not get 37 bytes. Allocators take pages from the kernel and cut them into size classes, so your 37-byte request lands in a 48-byte or 64-byte slot depending on the implementation. The difference is internal fragmentation: real, resident, and invisible from your code.

Allocators also keep their own metadata, and usually per-thread caches so threads are not fighting over a lock on every allocation. That is a deliberate trade of memory for throughput.

Resident memory is not object memory

Your process has a virtual address space, and the operating system maps parts of it to physical RAM. The number that matters for a capacity conversation is the resident set size, the pages actually in RAM right now. That includes your objects, but also code, stacks, thread caches, page tables and every buffer in flight.

This is why Cloudflare's benchmark entry shrank 56% while production process memory fell 43% at p99. Both numbers are honest; they measure different things. Micro-level byte math tells you the direction and the rough size of a win. Only production tells you the win.

What the containers cost, concretely

Now the layer where the five changes happen. I will use Rust names because that is what Cloudflare used, but map them onto whatever you write in.

Vec<T> and String are growable: pointer, length, capacity, plus a heap buffer with slack.

Box<[T]> and Box<str> are owned but fixed length: pointer and length, no capacity, and the buffer is exactly the size it needs. Same data, eight fewer bytes of metadata per field, no reserved slack.

Enums are the subtle one. A Rust enum is a tagged union: it can hold any one of several variants, so it must be big enough for the largest one, plus a discriminant, plus padding. Every value of that type pays the size of the worst case, even when it is holding the smallest variant.

enum RecordData {
    A([u8; 4]),        // 4 bytes of payload
    AAAA([u8; 16]),    // 16 bytes
    NAPTR(Naptr),      // the big one
}

If the largest variant is 136 bytes, the enum is around 144 bytes, and an A record holding four useful bytes still occupies all of it. Cloudflare reports exactly this: NAPTR at 136 bytes made the enum 144.

Option<T> is worth a note, because it is usually free. Rust can often store None in an impossible value of T rather than adding a flag byte, which is called a niche. A Box can never be null, so Option<Box<T>> is the same size as Box<T>. That matters for the third change below.

The five changes, byte by byte

Cloudflare's post walks through five successive representation changes. What makes the sequence instructive is that they are not independent tricks; each one exposed the next problem.

One: stop paying for growth you will never use

Once a DNS response is in the cache it is immutable. It will never append a record or a character. But it was built out of Vec and String, because that is what you naturally use while parsing.

The change is mechanical: Vec<T> becomes Box<[T]>, String becomes Box<str>. Cloudflare says the entry had eight such fields, so dropping the capacity word saved 8 bytes per field, 64 bytes per entry, plus whatever spare heap capacity those buffers were holding. It puts the combined saving from this change at over 15 TB across the fleet.

The general lesson is that data has phases, and the best representation is different in each one:

BUILDING -> FROZEN -> READ MANY TIMES -> EXPIRED

A builder wants to be mutable and forgiving. Storage wants to be compact and immutable. Conflating them means paying builder costs for the entire lifetime of the object. You already know this pattern under other names: a string builder that produces a string, a mutable B-tree page that gets written out as a compact immutable page, ingest rows that become a compressed columnar segment, a mutable graph that becomes a flat adjacency array before analytics.

The gotcha is real though. If updates turn out to be common after all, a frozen layout forces a copy on every change, and you have moved the cost rather than removed it. Decide from a measured lifecycle, not from taste.

Two: give several lists one allocation

A DNS response has three sections: answer, authority, additional. Three separate lists means three pointers, three lengths, three allocations.

The change stores one flat sequence and remembers where the boundaries are:

[ answer records | authority records | additional records ]
                 ^                   ^
                 offset 1            offset 2

Two u16 offsets replace two pointer and length pairs. Cloudflare puts that at 28 bytes per entry.

Two things are going on here, and only one of them is about bytes.

The obvious one is that a 16-bit integer is enough. A u16 covers 0 to 65535, and if the protocol or an enforced invariant guarantees section counts stay inside that, a 64-bit word is 6 bytes of nothing. This generalises: ports are 16 bits, status codes are nowhere near 64, a day of the week needs three bits. Just be careful, because a narrowed integer that can overflow is not an optimization, it is a correctness bug with a performance excuse. Narrow only where the domain gives you a hard bound.

The less obvious one is that three allocations became one. Three separate heap blocks can sit anywhere in memory. One block is contiguous, which means fewer cache line fills to walk it and less work for the allocator on every insert. That benefit shows up again in the performance section.

Three: store the value once and infer the rest

Most records in a DNS response have an owner name equal to the name that was queried, which is already the cache key. So the same name was stored once as the key and again on every record.

The change makes the owner optional and infers it:

cache key: example.com
record 1 owner: none        -> it is example.com
record 2 owner: none        -> it is example.com
record 3 owner: cdn.example.net   -> the exception, stored explicitly

This saves more than the length of a string. A domain name object can carry its own allocation, its own length, possibly a normalised form and label boundaries. Removing the duplicate removes the whole object, metadata included. And because Option<Box<T>> uses the null niche, the common case costs a pointer's worth of nothing.

The pattern is: store a value once at the parent scope, let children inherit it, and record only the exceptions. You have seen it in HPACK and QPACK header tables, in columnar dictionary encoding, in filesystem and database page prefix compression, in CSS inheritance, in config overlays, in protobuf defaults.

The cost is cognitive. The value is now implicit, which means every reader must have the context to reconstruct it. If one of these records ever escapes the scope that supplies the missing name, an absent field silently becomes wrong data rather than missing data. That is a worse failure than a crash, so the boundary where inference happens needs to be obvious in the code.

Four: make the common case cheap and the rare case expensive

Back to the 144-byte enum. Cloudflare reports that A records are 56% of its traffic and AAAA another 25%, so roughly four in five records need 4 or 16 bytes of payload while paying for 136.

The change boxes the large variants. Common small variants stay inline; rare large ones become a pointer to a heap allocation. Cloudflare puts the saving at 120 bytes per A or AAAA record.

The reasoning is statistical and worth internalising:

if 90% of values need 16 bytes and 10% need 136
inline everything:  always pay ~136
box the rare case:  pay ~16 for 90%, pay 16 + allocation for 10%

Expected memory falls a long way even though the rare case got slightly worse. This is the same instinct behind small-string optimization, tagged pointers, and inline-versus-spilled representations in every serious runtime.

And then it backfired, which is the part I like most about this sequence. Every box is another address to visit. Common records got smaller, but complex records scattered across the heap, and the lookup path now had more pointers to chase. Memory went down; locality got worse.

That is the rule this step teaches:

Count allocations, not just bytes.

Five: keep the bytes encoded

The last change is the architecturally interesting one.

The extreme version would be to cache the entire DNS response exactly as it goes on the wire, ready to send. Cloudflare did not do that, and the reason is instructive: different clients need different output. DNSSEC-related records depend on client flags, and name compression has to be decided fresh per response. A single prebuilt packet would mean either several variants or reparsing to filter, which trades the problem for a different problem.

The middle ground keeps the entry-level metadata structured, and stores the record data as one contiguous buffer where each record is a 2-byte length prefix followed by its raw wire-format bytes:

[len][raw record bytes][len][raw record bytes][len][raw record bytes]
  2B                     2B                     2B

That removes the large per-record enum, removes a box per rare record, removes most of the pointer graph, and removes serialization work for records that can be copied straight out. Cloudflare says A, AAAA, TXT and DNSSEC types can be copied directly, while records carrying domain names, such as CNAME, NS, MX and SOA, still need parsing so names can be compressed correctly. It attributes about 5% of the lookup latency improvement to this change alone.

The trade is that you lose random access. Records are variable length inside a byte stream, so reaching the fourth one means walking the first three.

Why smaller also turned out to be faster

Memory optimizations often cost speed, because the usual way to use less space is to compress and pay CPU to decompress. That is not what happened here, and the reason is worth making precise rather than waving at "better cache locality."

The hierarchy

A core reaches its data through levels, each larger and slower than the one before: registers, L1, L2, the shared last-level cache, then DRAM, then anything off the box. Exact latencies vary by CPU, but the ordering never does. Every level you avoid is time you do not spend stalled.

Cache lines

CPUs do not fetch bytes, they fetch cache lines, commonly 64 bytes. Suppose a lookup needs 32 bytes of record data.

If that data is contiguous, one line fill probably brings all of it, and the line is almost entirely useful.

If it is scattered across a pointer, a header, an owner name and a payload, the same answer needs four line fills, and most of each line is data you did not want. You moved 256 bytes to use 32.

Dependent loads

A CPU can keep many memory operations in flight, which is how it hides latency. It cannot do that when each address depends on the previous result:

load A -> only now do you know the address of B
       -> only now do you know the address of C

That chain serialises the stalls. Contiguous iteration gives the prefetcher predictable addresses and lets the machine overlap the work. This is the concrete meaning of "pointer chasing is slow", and it is why removing a level of indirection can help more than removing bytes.

Allocation is work

Insert throughput went up 43%, and lookups alone do not explain that. Building an entry with fewer allocations means less time finding size-class slots, less allocator metadata to touch, fewer trips to per-thread caches, fewer pages to acquire and zero.

Cloudflare's final design builds records in a reusable scratch buffer and then makes one exactly sized allocation for the packed bytes. The growth cost is amortised across inserts instead of paid per entry.

Work that stopped existing

The old representation held parsed records, so answering a query meant serializing them back into wire format. With the bytes already encoded, common types are a copy. That is not locality at all; it is instructions deleted from the hot path.

It helps to decompose lookup time rather than treat it as one number:

lookup = index work
       + memory fetches
       + decode and branching
       + response construction
       + serialization

This redesign reduced several terms at once, which is why attributing the whole 19% to any single cause would be wrong.

The machine this runs on

It is worth grounding all of this in the hardware, because "we have plenty of RAM" hides several distinct limits.

Cloudflare's published Gen 13 compute server is a single-socket 2U node: an AMD EPYC 9965 with 192 cores and 384 threads, 768 GB of DDR5-6400 across twelve fully populated memory channels for about 614 GB/s of peak bandwidth, three E1.S PCIe 5.0 NVMe drives for 24 TB, dual 100 GbE, and a 1300 W Titanium power supply.

Three things in that spec matter for this article.

Channels, not just capacity. "768 GB of DDR5" tells you how much fits, not how fast you can reach it. Bandwidth comes from the number of populated channels. A machine with the same capacity on half the channels has roughly half the bandwidth, and DIMM population is one of the easiest things to get wrong when a server is specified by capacity alone.

Bandwidth does not rescue latency. 614 GB/s is enormous, and it does nothing for code that follows one dependent pointer at a time. Streaming a contiguous array can saturate bandwidth. Chasing a pointer graph is latency-bound and leaves most of that capability idle. This is exactly why the layout change mattered on a machine that already had plenty of memory throughput.

Cores multiply the problem. With 384 hardware threads hitting the same cache, allocator lock contention and memory stalls are not rounding errors. Reducing allocations per insert is a concurrency improvement as much as a memory one.

Turning bytes into capacity

Here is the equation the whole thing rests on:

fleet cost of one byte = number of objects x 1 byte

At 250 billion objects:

1 byte  = 250 GB
4 bytes = 1 TB
8 bytes = 2 TB
64 bytes = 16 TB

Which is why removing one 8-byte capacity field from eight fields is a fleet-scale change before you count the reclaimed heap slack.

Now the honesty check. The benchmark shrank an entry by 533 bytes. Multiply by 250 billion and you get about 133 TB. Cloudflare reports about 100 TB. That gap is not an error in either number, and the reasons are worth knowing because you will hit the same gap:

• the entry count is dynamic and "over 250 billion" is a floor
• the benchmark's mix of entries is not exactly production's mix
• net footprint accounting is not the same as process resident memory
• caches do not all sit at the same fill level
• everything else in the process is unchanged
• allocators and pages do not hand back every logical byte one for one

The rule I would take from that: byte math predicts direction and order of magnitude. Production memory is an empirical measurement. Use the first to decide whether to do the work, and the second to report what the work achieved.

If you want to run this for your own system, the worksheet is short:

N = live objects per node
B = bytes saved per object
M = nodes
R = replication factor
H = an allocator and headroom factor, measured not guessed

logical bytes saved = N x B x M x R
expected RSS saved  = logical bytes saved x H

And only then, separately, the money question, which is a provisioning question and not a memory question:

servers avoided = ceil((capacity_needed_before - capacity_needed_after) / usable RAM per server)

Running this exercise yourself

The case study turns into a repeatable practice. Roughly this order.

Find the objects that actually dominate. Not the biggest objects, the biggest population. A 5 KB object with a thousand instances is 5 MB and irrelevant next to a 200-byte object with 500 million instances. Heap profiles, allocator statistics, resident memory and your own object counters will tell you which types those are.

Write the memory equation for each one before touching code. Inline struct size, plus every allocation it points at, plus average reserved capacity, plus allocator rounding, plus duplicated children, times observed cardinality. This step is accounting, and it usually reveals the answer before any refactoring happens.

Classify every field. This table is where the five changes come from, and you can build it for any type:

Question about the field	If yes, the move is
Does it stop changing after construction?	Drop the growth metadata, freeze it
Is the same value stored on a parent?	Store once, infer, record exceptions
Is it one of many tiny allocations?	Flatten into one allocation with offsets
Is it a rare huge variant of a union?	Box the rare case, inline the common one
Is it over-wide for its domain?	Narrow it, only against a hard bound
Is it decoded on write and re-encoded on read?	Keep it encoded
Is it a builder convenience on the hot path?	Split builder from frozen form

Count allocations per operation, not just bytes. Allocations and bytes per insert, then pointer dereferences and cache lines touched per lookup. Step four of the Cloudflare sequence is the reminder that these two can move in opposite directions.

Benchmark whole operations, and check the tail. An isolated function benchmark will happily show you a win that the full insert or lookup path never sees. And at infrastructure scale the number that hurts is p99, not the mean, because memory stalls and allocator slow paths land disproportionately in the tail.

Keep the builder. The pattern that makes all of this maintainable is two representations with an explicit conversion between them:

struct ResponseBuilder {
    // Vec, String, whatever is convenient while parsing
}

struct CachedResponse {
    // boxed slices, u16 offsets, packed bytes
}

The conversion point is where you validate the bounds you rely on and compact the data. You keep ergonomic mutation where mutation happens, and you stop paying for it everywhere else.

What to take from this

The reusable model is four questions about any representation:

1. How many bytes are useful payload?
2. How many are padding, container metadata and allocator rounding?
3. How many separate allocations does one logical object own?
4. In what order will the CPU touch them?

Cloudflare's 56% came from attacking unnecessary mutability, unnecessary duplication, unnecessary indirection and unnecessary decoding, in that order, measuring after each step and letting the result pick the next problem. The speedup came along because the changes removed work rather than trading space for time.

And the fleet part is the bit worth carrying furthest. Whether any of this is worth doing is not a question about elegance. It is a multiplication. Bytes times cardinality times lifetime is either a rounding error or a datacenter, and the same code change can be both depending on how many copies of the object exist.

Sources and further reading

The primary sources for every attributed number, and the material to go deeper.

• Cloudflare: How we saved 100 terabytes of memory by optimizing 1.1.1.1's DNS cache. The post this article reads. Every benchmark figure, per-optimization saving and production percentile above comes from here.
• Cloudflare: Inside Gen 13, our newest server generation. The hardware specification, including the memory channel configuration and bandwidth figure.
• The Rust Reference: Type layout. Size, alignment, field offsets, and why the default representation is not a stable contract. Read this before reasoning about any struct's size.
• Rust std: Vec and String. The pointer, length and capacity model that change one removes.
• Rust Performance Book: Type sizes by Nicholas Nethercote. Practical measurement of struct sizes, enum layout and boxing rare variants, with the tooling to see it.
• Google Research: Profiling a Warehouse-Scale Computer. Where cycles actually go across a fleet, and why memory hierarchy behaviour dominates so many server workloads.
• Google Research: Characterizing a Memory Allocator at Warehouse Scale. Allocation treated as a systems subsystem rather than an implementation detail.
• What Every Programmer Should Know About Memory by Ulrich Drepper. Long, old in places, and still the best single explanation of caches, DRAM and access patterns.
• Linux perf-stat manual. The counters that turn "it should be more cache friendly" into evidence.
• RFC 1035 for the DNS message format, if the sections and record structure above were new to you.

Continue learning

If the layout material landed, the next thing worth understanding properly is your allocator: size classes, thread caches, and how to read its statistics, because that is where the gap between logical bytes and resident bytes lives. After that, cache and memory behaviour at the hardware level, which turns the locality arguments here into numbers you can measure. Then data-oriented design as a discipline, where array-of-structs versus struct-of-arrays and hot-cold field splitting are the everyday tools. Those three cover most of what separates knowing an object is big from knowing why.

FAQ

Is this a Rust-specific result? No. Rust made the representation explicit and gave good tools for owned immutable data without a garbage collector. The ideas map onto C, C++, Go, Java, C# and database engines. Managed runtimes have their own object headers and layout rules, so the specific numbers differ while the accounting does not.

Did Cloudflare save money? Cloudflare did not publish a dollar figure, and turning freed memory into money requires assumptions about procurement, power, racks and timing that were not published either. What it did say is that it plans to use the freed memory for larger caches.

Should I go and box my large enum variants? Only after measuring. Boxing the rare variant is the right move when the distribution is skewed and memory is the constraint, and Cloudflare's own sequence shows it made locality worse before a later change fixed that. Count allocations alongside bytes.

How do I know if my objects are worth this effort? Multiply bytes saved by live object count by node count. If the answer is not at least a large fraction of a machine's memory, spend the week somewhere else.