I spent three weeks trying to run large language models locally on an RTX 3060 Ti. Every guide I found either assumed I had 24GB of VRAM or told me to buy a bigger GPU. Neither option was happening – 8GB is what I have, and I needed to make it work.
The result: I’m running a 35B parameter model with 128k context at 30-36 tokens per second on a single mid-range GPU. No cloud subscription. No API keys. No monthly fees. Just a local server that handles everything from code review to architectural planning.
Here’s exactly how I got there, step by step.
What You Need Before Starting
This setup works on any NVIDIA GPU with 8GB VRAM. I tested it on an RTX 3060 Ti, but a 3060, 4060, or any 8GB card should work. The key requirement is VRAM – system RAM helps but doesn’t substitute for it.
| Component | Minimum | My Setup |
| GPU VRAM | 8GB | RTX 3060 Ti 8GB |
| System RAM | 16GB | 32GB DDR4 |
| CPU | 4+ cores | Ryzen 7 5800X (8C/16T) |
| Storage | SSD | NVMe (fast model loading) |
| OS | Windows / Linux | Windows 11 |
You also need a model that uses MoE (Mixture of Experts) architecture. Dense models like a 13B or 34B will not fit in 8GB VRAM at usable context sizes. MoE models are the unlock – they activate only a fraction of their parameters per token, so you get the quality of a large model with the VRAM footprint of a small one.
Step 1: Install the Inference Engine
Standard llama.cpp works fine for most models. But for this specific setup, I use llama-cpp-turboquant – a fork that adds TurboQuant+ KV cache compression. This is the key piece that makes 128k context fit in 8GB.
Download the latest release for your platform:
- Windows: `turboquant-plus-tqp-v0.3.0-windows-x64-cuda12.4.zip`
- Linux: `turboquant-plus-tqp-v0.3.0-linux-x64-cpu.tar.gz`
The Windows build comes with CUDA 12.4 runtime DLLs bundled – no separate CUDA toolkit installation needed.
If you prefer to build from source:
# NVIDIA CUDA
cmake -B build -DGGML_CUDA=ON && cmake --build build -jThe TurboQuant+ quantization types become available automatically once the CUDA backend compiles.
Step 2: Download the Model
I use Ornith-1.0-35B-MTP-APEX-I-Mini – a Qwen3.5 MoE variant with 35B total parameters, 3B active per token, 256 experts, and multi-token prediction support.
Download the GGUF file directly:
# Using huggingface-cli
huggingface-cli download TheTom/Ornith-1.0-35B-MTP-APEX-I-Mini \
Ornith-1.0-35B-MTP-APEX-I-Mini.gguf \
--local-dir ./modelsOr download it manually from the HuggingFace model page and place it in a models/ directory.
The UD-IQ4_NL quantization is what I use – it’s aggressive enough to fit in VRAM but preserves enough quality for coding tasks. If you have headroom, Q4_K_M is slightly higher quality but uses more memory.
Step 3: The Server Command
This is the command that made everything work. Every flag matters:
llama-server \
-m Ornith-1.0-35B-MTP-APEX-I-Mini.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 999 \
-ncmoe 34 \
-fa on \
-ctk q8_0 \
-ctv turbo2 \
-c 128300 \
-b 512 \
-ub 2048 \
--no-mmap \
-t 5 \
-np 1Let me break down every flag because this is where people get lost:
Model Loading
| Flag | Value | Why |
| `-m` | model file | Path to the GGUF file |
| `-ngl` | 999 | Offload all transformer layers to GPU |
| `-ncmoe` | 34 | Number of MoE expert layers on GPU – this is the VRAM dial |
The -ncmoe flag is the most important one. It controls how many MoE expert layers live on the GPU versus CPU. At 34, the model fits in 8GB with room for the KV cache. At 35+, you start spilling to system RAM and speed drops. At 30, you have headroom but leave performance on the table.
KV Cache Compression
| Flag | Value | Why |
| `-ctk` | q8_0 | Keys at 8-bit – preserves attention quality |
| `-ctv` | turbo2 | Values at ~2-bit – aggressive compression where it’s safe |
This is the asymmetric KV cache trick. Keys need precision because they determine which tokens the model attends to. Values can be compressed harder because small errors in value space don’t degrade output as much. The turbo2 format uses Walsh-Hadamard rotated polar quantization – it’s not just rounding down to 2 bits, it’s a mathematically optimized codec that preserves quality at extreme compression.
The result: KV cache uses roughly 1/4 the memory of standard q8_0/q8_0, which is what frees enough VRAM to run 128k context.
Speculative Decoding (MTP)
| Flag | Value | Why |
| `–spec-type` | draft-mtp | Use multi-token prediction for speculative decoding |
| `–spec-draft-n-max` | 2 | Draft 2 tokens ahead – sweet spot for this model |
MTP lets the model predict the next 2 tokens, then verify them in parallel. When the draft is right (and it’s right 60-90% of the time depending on the task), you get 2 tokens for the cost of 1 verification pass. The actual speed gain depends on draft acceptance – I measured 25-36 t/s with this config versus 18-19 t/s without speculative decoding.
Performance Tuning
| Flag | Value | Why |
| `-fa` | on | Flash Attention – faster and uses less VRAM for long contexts |
| `-c` | 128300 | 128k context window |
| `-b` | 512 | Batch size for prompt processing |
| `-ub` | 2048 | Micro-batch size |
| `–no-mmap` | – | Prevents memory-mapped file issues on Windows |
| `-t` | 5 | CPU threads for the MoE routing layers |
| `-np` | 1 | Single parallel slot – freed ~600MB+ versus auto(4) |
The -np 1 flag is critical. By default, llama-server creates 4 parallel slots for concurrent requests. Each slot reserves its own KV cache, which eats VRAM. Since I’m running this as a single-user local server, I only need one slot. Setting -np 1 freed enough memory to increase the context window from 32k to 128k.
Step 4: Verify It Works
Start the server and check the output:
llama-server \
-m Ornith-1.0-35B-MTP-APEX-I-Mini.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 999 \
-ncmoe 34 \
-fa on \
-ctk q8_0 \
-ctv turbo2 \
-c 128300 \
-b 512 \
-ub 2048 \
--no-mmap \
-t 5 \
-np 1You should see initialization output showing:
- Model loaded successfully
- KV cache quantization: K=q8_0, V=turbo2
- Flash Attention enabled
- Context size: 128300
- Parallel slots: 1
Test it with a quick API call:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "ornith-35b",
"messages": [{"role": "user", "content": "Write a Python function to merge two sorted lists"}],
"temperature": 0.6
}'If you see tokens streaming back at 20-30 t/s, you’re good.
Step 5: Connect Your Coding Assistant
I use pi.dev as my coding assistant. It connects to the llama-server via the local API endpoint. Here’s my configuration:
{
"defaultProvider": "llama-server=http://127.0.0.1:8080",
"defaultModel": "Ornith-1.0-35B-MTP-APEX-I-Mini.gguf",
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20,
"repeat_penalty": 1.02,
"defaultThinkingLevel": "high"
}The repeat penalty of 1.02 prevents the model from getting stuck in output loops without suppressing useful repetition. Temperature 0.6 keeps coding outputs focused. And “high” thinking level lets the model reason through complex architectural problems instead of rushing to an answer.
If you use a different tool – VS Code with Continue, Zed, or any OpenAI-compatible client – point it at http://127.0.0.1:8080 and it will work. The llama-server exposes a standard OpenAI-compatible API.
What This Setup Actually Gives You
After three weeks of testing, here’s what running a 35B model at 128k context on 8GB VRAM looks like in practice:
| Metric | Value |
| Generation speed | 30-36 t/s |
| Prompt ingestion | 670-720 t/s |
| Context window | 128k tokens (~500k characters) |
| Draft acceptance | 62-90% depending on task |
| VRAM usage | ~7.6-8GB (maxed out) |
| Model loading time | ~15-20 seconds |
The 128k context means I can feed it an entire codebase – multiple files, configuration, documentation – and it holds everything in working memory. For a WordPress developer working with Laravel, Vue, and custom plugins, this is the difference between “explain this one function” and “review this entire module and suggest improvements.”
The generation speed of 30-36 t/s is fast enough that I never wait. It’s not GPT-4 speed, but for local inference on a mid-range GPU, it’s more than usable. Most of the time I’m reading the output as fast as it generates.
What Didn’t Work (And Why)
Before I landed on this config, I tried several approaches that failed:
Standard llama.cpp with q8_0/q8_0 KV cache: Ran out of VRAM at 32k context. The KV cache for a 35B model at 128k with symmetric 8-bit quantization needs ~12GB – impossible on 8GB cards.
Vulkan backend with Gemma 4 E4B: Got 8 t/s. Vulkan with auto parallel slots is not competitive with CUDA for local inference.
Qwen3-Coder 30B at low MoE offload (20 layers): 14 t/s. Too slow to be useful. The model needs at least 37 MoE layers on GPU to hit acceptable speeds.
Draft-n=4 with Qwen3.6 MTP: Crashed. Too many draft tokens create more overhead than they save. n=1 or n=2 is the sweet spot.
Running multiple models simultaneously: VRAM contention kills performance. One model at a time, loaded on demand.
The lesson: VRAM is a hard resource constraint. Every flag in the server command exists to optimize within that constraint. Remove one and something breaks – either speed drops, context shrinks, or the model spills to CPU and becomes unusable.
The Stack That Emerged
From this testing, I settled on a three-model rotation:
- Gemma 4 E4B at 68 t/s for quick tasks
- Qwen2.5-Coder 7B at 37 t/s for daily coding
- Ornith 35B at 30-36 t/s for hard problems
Each model loads on demand. Only one runs at a time. The total VRAM never exceeds 8GB. And the workflow feels like having a fast intern, a solid mid-level developer, and a senior architect – all running locally, all free, all private.
If you’ve been told you need a 4090 to run local AI for coding, you’ve been told wrong. You need the right model architecture, the right quantization, and the right flags. That’s it.
The setup works. I use it daily. And now you have every command to make it work on your machine too.
