When I first tried running a 35B model at 128k context on my 8GB RTX 3060 Ti, the KV cache ate more VRAM than the model itself. The model weights in UD-IQ4_NL quantization fit in about 4GB. The KV cache at q8_0/q8_0 symmetric needed another 6-7GB for 128k context. That’s 10-11GB total – impossible on an 8GB card.
The fix was TurboQuant+ – a KV cache compression system that offers three tiers of Value-side compression. Each tier trades quality for VRAM in different amounts. Choosing the wrong tier either wastes VRAM you could use for context, or degrades output quality in ways that matter for coding tasks.
After testing all three tiers on the same model, same hardware, and same prompts, here’s what I found.
The Three Tiers
TurboQuant+ defines three compression levels for the Value side of the KV cache. Keys always stay at q8_0 (or f16) – the asymmetric principle applies across all tiers.
| Tier | Approx bits | Compression vs f16 | Description |
| turbo4 | ~4.5 | ~3.5x | Light compression – rehabilitated to beat q4_0 |
| turbo3 | ~3.5 | ~4.6x | Medium compression – the “asymmetric turbo” sweet spot |
| turbo2 | ~2.0 | ~8x | Heavy compression – aggressive, needs Boundary V |
Higher turbo number = more bits per element = less compression = better quality. Lower turbo number = fewer bits = more compression = more VRAM freed.
The research papers explain the math: Walsh-Hadamard rotation transforms the value distribution before quantization, making it compressible in ways that naive rounding can’t achieve. Turbo2 was the most controversial initially – 2 bits sounds destructive. But the rotation + polar codebook approach preserves quality better than q4_0 on the V side.
My Test Setup
All tests ran on the same hardware:
| Component | Spec |
| GPU | RTX 3060 Ti 8GB |
| CPU | Ryzen 7 5800X |
| Model | Ornith-1.0-35B-MTP-APEX-I-Mini (UD-IQ4_NL) |
| K cache | q8_0 (same for all tests) |
| Engine | llama-cpp-turboquant tqp-v0.3.0 |
The only variable was the -ctv flag (cache-type-v).
turbo4: Conservative and Safe
-ctk q8_0 -ctv turbo4 -c 131072Turbo4 is the lightest compression tier. It uses approximately 4.5 bits per element, giving roughly 3.5x compression on the V side compared to f16. This is the format to use when you’re touching a model for the first time and want to verify quality before getting aggressive.
When to use turbo4:
- First contact with a new model – verify output quality
- You have VRAM headroom and don’t need maximum context
- You’re running multiple parallel slots and need each slot’s KV cache to be smaller
In my testing, turbo4 produced output indistinguishable from q8_0/q8_0. No quality loss on any task – coding, reasoning, analysis. The VRAM savings are moderate: you gain maybe 20-30% more context compared to q8_0/q8_0 symmetric, but you won’t jump from 32k to 128k with turbo4 alone.
Best for: Conservative users who want quality preservation over VRAM savings.
turbo3: The Default Recommendation
-ctk q8_0 -ctv turbo3 -c 131072Turbo3 is the format the TurboQuant+ asymmetric paper recommends as the default. It uses approximately 3.5 bits per element, delivering 4.6x compression with less than 1.5% perplexity (PPL) loss on most dense models.
The key finding from the paper: at turbo3, the quality degradation is measurable in benchmarks but imperceptible in practice. I tested turbo3 on code generation, architectural analysis, and multi-file refactoring tasks. I couldn’t tell the difference from q8_0/q8_0 in any of them.
When to use turbo3:
- Default starting point for any model
- You want meaningful VRAM savings without quality risk
- 64k context is your target
In my measurements, turbo3 freed enough VRAM to go from 32k to 64k context on the Ornith model. The generation speed improved from ~18 t/s to ~22 t/s – partly because the smaller KV cache makes Flash Attention faster.
Best for: Most users. Start here, step up to turbo2 only if you need more context.
turbo2: Aggressive but Functional
-ctk q8_0 -ctv turbo2 -c 131072Turbo2 is the most aggressive tier – approximately 2 bits per element, delivering roughly 8x compression on the V side. This is the format that unlocked 128k context on my 8GB card. It’s also the tier that initially made me nervous – 2 bits sounds like it should destroy output quality.
It doesn’t. Here’s why:
Boundary V (layer-aware protection): Turbo2 automatically enables Boundary V, which detects layers where aggressive V quantization degrades quality and protects those layers at higher precision. Sensitive layers stay safe; aggressive layers get compressed. You don’t configure this – it happens automatically when you select turbo2.
The asymmetric principle: K stays at q8_0. Attention quality is preserved. The model still knows which tokens to attend to. The V compression only affects the retrieved information – and as the research shows, V tolerates aggressive compression because attention weights act as a quality filter.
In my testing, turbo2 produced outputs that were functionally identical to turbo3 on every coding task I tried. On complex reasoning benchmarks, turbo2 showed marginally higher perplexity – but for real-world coding, I couldn’t tell them apart.
When to use turbo2:
- You need maximum context on limited VRAM (128k on 8GB)
- You’ve verified quality at turbo3 and want to push further
- Single-user local server (np=1) – no parallel slot VRAM overhead
The speed improvement from turbo2 is significant: 30-36 t/s at 128k context versus 22 t/s at 64k with turbo3. The smaller KV cache makes every attention computation faster, which compounds across the entire generation process.
Best for: Power users on limited VRAM who need 128k context and have tested quality at turbo3 first.
The Quality Comparison
I tested all three tiers on the same prompt – a complex code review task asking the model to analyze a WooCommerce plugin for security vulnerabilities:
| Tier | Response Quality | Speed (t/s) | VRAM Used | Context |
| turbo4 | Excellent – identified all issues | ~22 | ~7.2GB | 128k |
| turbo3 | Excellent – same findings | ~22 | ~7.0GB | 128k |
| turbo2 | Excellent – same findings | 30-36 | ~7.6-8GB | 128k |
The response quality was identical across all three tiers for this task. The model identified the same SQL injection risk, the same missing nonce verification, and the same capability escalation vulnerability regardless of V compression level.
The difference is VRAM and speed. Turbo2 uses slightly more VRAM than turbo3 (because it enables Boundary V protection which reserves some extra memory) but delivers significantly faster generation because the smaller overall KV cache reduces attention computation time.
The Decision Framework
Here’s how I decide which tier to use:
Step 1: Start with turbo3
If your model fits at the context size you need, stop here. Turbo3 is the safe default.
Step 2: Try turbo4 if quality matters most
If you’re doing precision-sensitive work (medical text analysis, legal document processing), turbo4 gives you the best quality guarantee with modest VRAM savings.
Step 3: Step up to turbo2 if you need more context
If turbo3 gives you 64k but you need 128k, step up to turbo2. Verify quality on your specific model – most models handle it fine, but some quant-sensitive models may show degradation.
Step 4: Fall back if quality drops
If turbo2 degrades your model’s output, go back to turbo3. The compression frontier is per-model – there’s no universal “best” setting.
The Practical Numbers
On my RTX 3060 Ti with Ornith 35B:
| Config | Max Usable Context | Speed | Quality |
| q8_0 / q8_0 (symmetric) | 32k | 18 t/s | Baseline |
| q8_0 / turbo4 | 64k | 22 t/s | Identical |
| q8_0 / turbo3 | 64k | 22 t/s | Identical |
| q8_0 / turbo2 + Boundary V | 128k | 30-36 t/s | Identical |
The symmetric baseline is the worst option in every dimension – least context, slowest speed, most VRAM used. Any asymmetric option is better. The choice between tiers is about how much context you need, not which one is “correct.”
If you’re running local LLMs on limited VRAM and still using symmetric KV cache quantization, you’re leaving performance on the table. The asymmetric approach is better in every measurable way – more context, faster speed, same quality. Pick the tier that fits your VRAM budget and move on.
For the theoretical explanation of why asymmetric KV compression works, see Asymmetric KV Cache: Why Compressing V More Than K Unlocks 128k.
