Asymmetric KV Cache: Why Compressing V More Than K Unlocks 128k

Every local LLM guide tells you to pick a quantization level for your model weights. Almost none of them mention the KV cache – the memory structure that stores every token the model has seen in the current conversation. On limited VRAM, the KV cache is what kills your context window. And the solution is counterintuitive: compress the K and V sides differently.

I discovered this the hard way. My first attempt at running a 35B model at 128k context crashed immediately – the KV cache alone needed more VRAM than my entire GPU had. The fix wasn’t a smaller model or a more aggressive weight quantization. It was asymmetric KV compression: keeping Keys at high precision while compressing Values aggressively.

Here’s why that works, and how to set it up.

What the KV Cache Actually Does

When a language model processes text, it doesn’t just look at the current token. It looks at every previous token in the context window. To avoid recomputing attention scores from scratch every time, it stores intermediate results in two matrices:

K (Key): Determines which previous tokens are relevant to the current token. When the model generates the next word, it compares the current token’s query against all stored keys to decide which past tokens to attend to.

V (Value): Contains the actual information from each previous token. Once the attention weights are calculated (using K), the model multiplies those weights by V to get the final context representation.

Think of it like a search engine. K is the index – it determines what you find. V is the content – it’s what you get when you find it. The index needs to be precise or you’ll retrieve the wrong things. The content can tolerate some noise because small errors in retrieved information rarely change the overall meaning.

This is the fundamental asymmetry that makes different compression levels possible.

Why Symmetric Compression Fails

The intuitive approach is to compress K and V equally. Use q8_0 for both. Use q4_0 for both. Keep them symmetric. This is what standard llama.cpp does.

The problem is that K and V have completely different sensitivity to quantization error:

K is fragile: A small error in a Key vector can change which tokens the model attends to. If the model thinks token #427 is relevant when it should be attending to token #1893, the output quality degrades. Quantization error in K directly causes attention misalignment.

V is robust: A small error in a Value vector means the retrieved information is slightly noisier, but the overall meaning is preserved. If the model retrieves “use Laravel’s auth middleware” but the quantization adds a tiny bit of noise, the output is still correct. The attention weights act as a quality filter – even if V is noisy, only the most relevant tokens contribute significantly.

When you compress both equally to q4_0, you’re being unnecessarily aggressive on K (causing attention errors) while being unnecessarily conservative on V (wasting VRAM on precision you don’t need).

The Asymmetric Solution

The TurboQuant+ research documents this finding explicitly: compress V much harder than K, and you get nearly the same quality at a fraction of the memory cost.

My configuration:

The turbo2 format isn’t just rounding values to 2 bits. It uses Walsh-Hadamard rotation followed by polar codebook quantization on 128-element blocks. The rotation transforms the value distribution into something that compresses much more efficiently than naive quantization. The research paper explains why MSE-driven codecs fail for KV compression – the errors compound in attention space, not value space.

The result: my 35B model’s KV cache shrank by roughly 4x, freeing enough VRAM to go from 32k context to 128k context. Same GPU. Same model. Same speed. Just smarter compression.

Boundary V: Layer-Aware Protection

The turbo2 format has a further optimization called Boundary V. Not all transformer layers are equally sensitive to V compression. Some layers – typically the early and late layers – are more delicate. Boundary V automatically detects which layers need protection and applies aggressive turbo2 compression only to the layers that can handle it, while leaving sensitive layers at higher precision.

This happens automatically when you use -ctv turbo2. The server detects the layer structure and applies the policy. You don’t need to configure anything. The layer-aware compression paper explains the specific per-layer sensitivity analysis – certain attention heads are more fragile than others, and Boundary V maps this automatically during model initialization.

In my testing, Boundary V was the difference between turbo2 being usable and turbo2 being great. Without it, I saw occasional quality degradation on complex reasoning tasks – the model would produce slightly less coherent explanations for multi-step debugging problems. With Boundary V enabled, the output was indistinguishable from q8_0/q8_0 at any context size I tested. The MoE-specific research extends this further for mixture-of-experts models, where different expert layers have different compression tolerances.

The practical impact: Boundary V uses about 200-300MB more VRAM than turbo2 without Boundary V, because it reserves extra memory for the protected layers. But that investment pays for itself in quality preservation. If you’re using turbo2, always leave Boundary V enabled – it’s the default behavior and there’s no reason to disable it.

How to Set It Up

If you’re using llama-cpp-turboquant (the fork that includes TurboQuant+), the setup is two flags:

llama-server \\
  -m model.gguf \\
  -ctk q8_0 \\
  -ctv turbo2 \\
  -c 131072 \\
  ...

That’s it. -ctk q8_0 sets the Key cache to 8-bit. -ctv turbo2 sets the Value cache to turbo2 compression with automatic Boundary V protection.

If you’re using standard llama.cpp, the closest equivalent is:

llama-server \\
  -m model.gguf \\
  -ctk q8_0 \\
  -ctv q4_0 \\
  -c 32768 \\
  ...

The q4_0 V cache is less aggressive than turbo2 – you won’t get 128k context on 8GB, but you’ll get more context than symmetric q8_0/q8_0. The compression ratio matters more than the exact format at small VRAM budgets.

The Compression Ladder

TurboQuant+ offers three compression levels for the V side, from lightest to most aggressive:

My recommendation: start at turbo3. If you have VRAM headroom, you’re done. If you need more context, step up to turbo2. If turbo2 degrades quality on your specific model, fall back to turbo3.

The compression ladder is per-model. A model that works perfectly at turbo2 might show quality issues at turbo3 on a different architecture. There’s no universal “best” setting – you test, you compare, you pick what works.

What I Measured

Here’s the concrete impact of asymmetric KV on my RTX 3060 Ti running Ornith 35B:

The symmetric config maxes out at 32k because the KV cache consumes too much VRAM. Asymmetric turbo3 doubles the context to 64k with a speed improvement. Turbo2 doubles it again to 128k – and the speed actually improves further because the model has more context to work with, which reduces attention re-computation.

The speed improvement isn’t just about fitting in VRAM. When the KV cache is smaller, Flash Attention operates more efficiently. The attention computation scales with KV cache size, so a 4x smaller cache means faster attention passes across the board.

Here’s what those context sizes mean in practice. At 32k tokens, you can hold roughly 25,000 words of conversation – maybe 10-15 exchanges with a coding assistant before you hit the limit. That’s enough for a short debugging session, but not enough for a full code review of a medium-sized file. At 64k tokens, you get 50,000 words – enough for a thorough code review plus conversation history. At 128k tokens, you get 100,000 words – enough to load an entire codebase into context and still have room for a detailed conversation about architecture changes.

The jump from 32k to 128k isn’t just a number. It’s the difference between “I can ask about this one function” and “I can show the model the entire module, the database schema, the API routes, and the test suite, and ask it to find the bug.” That’s a fundamentally different workflow. You stop context-switching between files and start working with the full picture.

The Takeaway

If you’re running local LLMs on 8GB VRAM and hitting context limits, the KV cache is your bottleneck – not the model weights. Asymmetric compression (precise K, aggressive V) is the single highest-impact optimization you can make. It’s the difference between 32k and 128k context on the same hardware.

The TurboQuant+ research proved this with formal papers and benchmarks. My own testing confirms it: 30-36 t/s at 128k context on an RTX 3060 Ti. No other optimization comes close to this kind of improvement.

Stop compressing K and V equally. The math doesn’t support it, and your VRAM budget can’t afford it.

If you’re starting from scratch, here’s the order of operations. First, check if your inference engine supports asymmetric KV compression – standard llama.cpp supports -ctk and -ctv flags, but turbo2 and Boundary V require the turboquant fork. Second, start with q8_0 for K and turbo3 for V as your baseline. Run your typical workload and check for quality degradation. If quality is fine, try stepping up to turbo2. If turbo2 shows quality issues on your specific model, fall back to turbo3. The whole process takes about 20 minutes of testing, and the payoff is a 4x larger context window for free.

For the benchmark numbers comparing each compression tier, see turbo2 vs turbo3 vs turbo4: Which KV Cache Quantization Fits Your VRAM.