Every “best local LLM” article in 2026 tells you what to download. None of them show you the actual numbers. They say “this model is fast” without telling you how fast on YOUR hardware, with YOUR quantization, at YOUR context length.
I got tired of guessing. So I ran 32 benchmark tests across 9 different models on my RTX 3060 Ti, recorded every number that mattered, and compiled the full dataset here. No cherry-picking. No rounding up. Just raw performance data from an 8GB VRAM card that most developers actually own.
If you’re trying to run local AI for coding on consumer hardware, this is the reference you wish existed.
My Hardware and Why It Matters
The specs that determine whether a local LLM runs well or runs at all:
| Component | Spec |
| GPU | NVIDIA RTX 3060 Ti – 8GB GDDR6 |
| CPU | AMD Ryzen 7 5800X – 8 cores / 16 threads |
| RAM | 32GB DDR4 |
| Storage | NVMe SSD (model loading speed matters) |
| OS | Windows 11 |
| Inference engine | [llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant) (fork of llama.cpp with TurboQuant+ KV cache compression) |
The RTX 3060 Ti is the most common mid-range GPU among developers. It’s not a 4090. It’s not a 3090 with 24GB of VRAM. It has 8GB, and that’s the hard ceiling. Every model, every quantization choice, every context window has to fit within that 8GB or it spills to system RAM and inference turns into a slideshow.
This is why I tested everything on this specific card. If it works here, it works on most setups.
Why MoE Models Changed Everything on 8GB
Before I get into the numbers, you need to understand why some models punch way above their weight class on limited VRAM.
Traditional dense models use all their parameters for every token. A 13B model activates all 13 billion parameters for every single token it generates. That means the entire model has to fit in VRAM.
MoE (Mixture of Experts) models work differently. A 35B model like Ornith or Qwen3.6 35B-A3B only activates a fraction of its parameters per token – typically 3B out of 35B total. The model has 256 experts, picks 8 per token, and only those 8 fire.
The result: you get the reasoning quality of a 35B model with the VRAM footprint closer to a 3B model. On 8GB VRAM, this is the unlock that makes everything else possible.
The 9 Models I Tested
I tested models across four categories: small fast workers, medium daily drivers, large reasoning models, and one failure. Here’s the lineup:
Small Fast Workers
Gemma 4 E4B (Q4_K_M quantization)
- Google’s small MoE model – 4B active parameters
- Ran at 64k context, CUDA backend
- This was the speed champion of the entire benchmark
Qwen3-4B Thinking (Q4_1 quantization)
- Alibaba’s small thinking model
- Designed for reasoning tasks despite its size
- Good for simple planning tasks
Medium Daily Drivers
Qwen2.5-Coder 7B (Q4_K_M quantization, 128k context)
- The dedicated coding model from Alibaba
- 7B dense parameters – fits comfortably in VRAM
- This is what I reach for most coding tasks now
Qwen3.5 9B (Q4_K_M quantization)
- General-purpose model with vision capabilities
- Tested at 125k context – it worked but the context pressure was real
Large Reasoning Models
Gemma 4 26B-A4B (Q4_K_M and UD-IQ4_NL quantizations)
- Google’s larger MoE – 26B total, 4B active per token
- Tested at both 4k and 64k context with different MoE layer offload counts
- Solid reasoning, slower generation
Qwen3-Coder 30B-A3B (IQ4_XS quantization)
- Alibaba’s large coding MoE – 30B total, 3B active
- Tested extensively with MoE layer offload from 20 to 41 layers
- Interesting but not the daily driver I hoped for
Qwen3.6 35B-A3B MTP (UD-IQ4_NL quantization)
- The most advanced Qwen model with multi-token prediction (MTP)
- Tested at 32k and 128k context
- MTP with draft-n=1 was the best 32k performer
Ornith-1.0-35B-MTP-APEX-I-Mini (UD-IQ4_NL quantization)
- Qwen3.5 MoE variant – 35B total, 3B active, 256 experts (8 active per token)
- The model I ultimately settled on
- Stable at 128k context with turbo2 KV cache compression
The Failure
GLM-4.6V-Flash (Q4_K_M + mmproj)
- Zhipu’s multimodal model
- Tested three times, rejected every time – bad output quality despite decent speeds
The Full Benchmark Data
Every row is a real test run. No averaging, no estimates. The numbers are what the server reported.
Gemma 4 E4B
| Config | Context | Prompt t/s | Gen t/s | Draft Acceptance | Verdict |
| Q4_K_M, CUDA, q8 KV, 8 threads | 64k | 1664 | 68.17 | – | Keep |
| Q4_K_M, CUDA, q8 KV, 8 threads | 64k | 2346 | 62.37 | – | Keep |
| Q4_K_M, Vulkan, default slots | 4096 | 2.98 | 8.03 | – | Reject config |
The CUDA config hit 68 t/s generation – the fastest result in the entire benchmark. Vulkan with auto slots was unusable at 8 t/s. Lesson: always use CUDA, never Vulkan for this model.
Qwen3-4B Thinking
| Config | Context | Prompt t/s | Gen t/s | Verdict |
| Q4_1, CUDA, q4 KV | 64k | – | 86.00 | Keep |
Fast but limited use case. Good for quick planning, not detailed coding.
Qwen3.5 9B
| Config | Context | Prompt t/s | Gen t/s | Verdict |
| Q4_K_M, CUDA, q4 KV | 125k | – | 22.58 | Maybe |
Works at massive context but the speed suffers. Not a daily worker.
Gemma 4 26B-A4B
| Config | MoE layers | Context | Prompt t/s | Gen t/s | Verdict |
| Q4_K_M, q8 KV | auto | 4096 | 19.40 | 13.79 | Keep (baseline) |
| Q4_K_M, q8 KV | 25 | 64k | 40.13 | 19.07 | Keep |
| Q4_K_M, q8 KV | 24 | 64k | 80.15 | 19.28 | Keep |
| Q4_K_M, q8 KV | 22 | 64k | 84.38 | 18.34 | Keep |
| UD-IQ4_NL, q8 KV | 41 | 72k | 43.59 | 18.10 | Keep |
| UD-IQ4_NL, q8 KV | 30 | 72k | 360.74 | 17.12 | Keep |
| UD-IQ4_NL, q8 KV | 29 | 72k | 381.51 | – | Partial (truncated) |
The Gemma 26B is a workhorse. Prompt processing is excellent at higher MoE layer counts. Generation stays around 17-19 t/s regardless of config – consistent but not fast.
Qwen3-Coder 30B-A3B
| Config | MoE layers | Context | Prompt t/s | Gen t/s | Verdict |
| IQ4_XS, q8 KV | 20 | 32k | 19.21 | 14.71 | Maybe |
| IQ4_XS, q8 KV | 25 | 32k | 109.57 | 9.97 | Maybe |
| IQ4_XS, q8 KV | 30 | 32k | 116.69 | 12.66 | Maybe |
| IQ4_XS, q8 KV | 36 | 32k | 40.07 | 19.62 | Keep |
| IQ4_XS, q8 KV | 37 | 32k | 361.89 | 18.57 | Keep |
| IQ4_XS, q8 KV | 40 | 32k | 268.75 | 17.76 | Keep |
| IQ4_XS, q8 KV | 41 | 72k | 39.32 | 17.53 | Keep |
| IQ4_XS, q8 KV | 37 | 128k | 20.35 | 16.71 | Maybe |
The Qwen3-Coder 30B was my attempt at a dedicated local coding model. At 37 MoE layers it’s solid – 18.57 t/s generation with 361 t/s prompt processing. But at 128k context it drops to 16.71 t/s. Compared to the Ornith model, it’s not competitive enough to justify the complexity.
Qwen3.6 35B-A3B MTP
This is where things got interesting. The Qwen3.6 has multi-token prediction (MTP) – it can draft multiple tokens ahead and speculative-decode them in parallel.
| Config | MoE layers | MTP n | Context | Prompt t/s | Gen t/s | Draft Acceptance | Verdict |
| UD-IQ4_NL, q8 KV | 45 | 2 | 32k | 16.21 | 25.05 | 84.58% | Keep |
| UD-IQ4_NL, q8 KV | 45 | 2 | 32k | 52.29 | 21.27 | 60.01% | Keep |
| UD-IQ4_NL, q8 KV | 33 | 2 | 32k | 335.65 | 26.24 | 62.69% | Keep |
| UD-IQ4_NL, q8 KV | 33 | 1 | 32k | 361.67 | 27.19 | 74.82% | Best 32k |
| UD-IQ4_NL, q8 KV | 33 | 4 | 32k | 302.92 | ~11-12 | – | Reject (too aggressive) |
| UD-IQ4_NL, q8 KV | 33 | 3 | 32k | 320.05 | ~21-22 | – | Partial |
| UD-IQ4_NL, q8 KV | 37 | 1 | 128k | 346.39 | 24.66 | 73.28% | Best 128k |
The sweet spot was draft-n=1 – 27.19 t/s generation with 74.82% draft acceptance at 32k context. Draft-n=4 was too aggressive and killed performance. Draft-n=2 was slightly faster acceptance (84%) but slower generation. n=1 is the winner.
Ornith-1.0-35B-MTP-APEX-I-Mini
The final model. The one I run daily now.
| Config | MoE layers | Context | Prompt t/s | Gen t/s | Draft Acceptance | Verdict |
| UD-IQ4_NL, q8 K + turbo2 V | 34 | 128k | 670-720 | 30-36 | 62.7% – 90.3% | Best overall |
The Ornith model with turbo2 KV cache compression on the V side is the winner across the board. 30-36 t/s generation at 128k context, prompt ingestion at 670-720 t/s, and draft acceptance hitting 90%+ on optimized runs.
The key was the TurboQuant+ asymmetric KV cache – q8_0 for Keys (preserving attention quality) and turbo2 for Values (aggressive compression where it’s safe). This freed enough VRAM to run the full model at 128k context without spilling to system RAM.
GLM-4.6V-Flash
| Config | Context | Gen t/s | Verdict |
| Q4_K_M + mmproj, CUDA | 32k | 9.75 | Reject |
| Q4_K_M + mmproj, CUDA | 32k | 12.31 | Reject |
| Q4_K_M + mmproj, CUDA | 32k | 18.91 | Reject |
Three attempts. All rejected. The model outputs garbled Chinese text despite being prompted in English, throws tokenizer warnings, and the quality never improved regardless of speed. Some models just don’t work – and that’s useful data too.
The Top 5 Results, Ranked
| Rank | Model | Speed (t/s) | Context | Best Use Case |
| 1 | Ornith 35B-A3B | 30-36 | 128k | Senior planner, hard coding tasks |
| 2 | Qwen3.6 35B MTP (n=1) | 27.19 | 32k | Planning, code review |
| 3 | Gemma 4 E4B | 68.17 | 64k | Fast general worker |
| 4 | Qwen2.5-Coder 7B | 37.27 | 128k | Daily coding assistant |
| 5 | Gemma 4 26B-A4B | 18-19 | 72k | Reasoning, analysis |
How I Use These Results Daily
I don’t run all 9 models. I run three, each assigned to a specific role:
Gemma 4 E4B – my fast worker. 68 t/s for quick tasks: file summaries, simple refactors, generating boilerplate. It’s fast enough that I never wait.
Qwen2.5-Coder 7B – my daily coder. 37 t/s with 128k context. It actually understands code structure, handles multi-file edits, and the context window is massive enough for entire codebases.
Ornith 35B-A3B – my senior planner. 30-36 t/s at 128k context. I use this for architectural decisions, complex debugging, code review, and any task where I need the model to think deeply before answering.
These three models cover every coding scenario I encounter. The total VRAM footprint stays within 8GB because only one model loads at a time, and the MoE architecture keeps the active parameter count low.
How to Reproduce These Benchmarks
If you want to run the same tests on your hardware, here’s the template command I used:
llama-server \
-m Ornith-1.0-35B-MTP-APEX-I-Mini.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 999 \
-ncmoe 34 \
-fa on \
-ctk q8_0 \
-ctv turbo2 \
-c 128300 \
-b 512 \
-ub 2048 \
--no-mmap \
-t 5 \
-np 1Key flags explained:
| Flag | What it does |
| `-ngl 999` | Offload all layers to GPU |
| `-ncmoe 34` | Number of MoE expert layers on GPU |
| `-ctk q8_0` | KV cache Key quantization – q8_0 preserves attention quality |
| `-ctv turbo2` | KV cache Value quantization – aggressive compression, safe for V |
| `-c 128300` | Context size (~128k tokens) |
| `-np 1` | Single parallel slot – freed ~600MB+ VRAM vs auto(4) |
| `–spec-type draft-mtp` | Enable multi-token prediction speculative decoding |
| `–spec-draft-n-max 2` | Draft 2 tokens ahead (sweet spot for this model) |
For my pi.dev setup, I pair this server with these generation settings:
{
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20,
"repeat_penalty": 1.02,
"defaultThinkingLevel": "high"
}The repeat penalty of 1.02 is light enough to prevent loops without killing the model’s creativity. Temperature 0.6 keeps outputs focused for coding tasks. And “high” thinking level lets the model reason through complex problems properly.
The Data Is the Point
I’m not going to tell you which model is “best” – your hardware, your workflow, your context needs are different from mine. What I will tell you is that most local AI content online is opinion without evidence.
These 32 test runs are evidence. Use them as a starting point, run your own tests, and find what works on your specific setup. The numbers don’t lie, but they also don’t transfer between GPUs. Your 4070 will be different from my 3060 Ti. Your 16GB setup will open options my 8GB card doesn’t have.
Start with the data. Make your own decision. And stop reading articles that tell you what to download without showing you why.
If you found this useful, the full benchmark dataset is something I’ll keep updating as I test new models. The local AI space moves fast – what’s slow today might be fast tomorrow with a new quantization trick or a model architecture that fits better in 8GB.
