{"id":1768,"date":"2026-07-16T09:30:00","date_gmt":"2026-07-16T09:30:00","guid":{"rendered":"https:\/\/nassimstudio.com\/blog\/?p=1768"},"modified":"2026-07-22T13:56:53","modified_gmt":"2026-07-22T13:56:53","slug":"ornith-35b-8gb-vram-128k-context-benchmarks","status":"publish","type":"post","link":"https:\/\/nassimstudio.com\/blog\/ornith-35b-8gb-vram-128k-context-benchmarks\/","title":{"rendered":"Ornith 35B on 8GB VRAM: 128k Context at 30 t\/s"},"content":{"rendered":"<p>I tested every model that claims to be good for local coding. Gemma 4, Qwen3-Coder, Qwen3.6 MTP &#8211; all of them. The model that actually earned a permanent spot on my machine is one most people haven&#8217;t heard of: <a href=\"https:\/\/huggingface.co\/Tribbler\/ornith-1.0-apex\" target=\"_blank\" rel=\"noopener\">Ornith-1.0-35B-MTP-APEX-I-Mini<\/a>.<\/p>\n<p>It&#8217;s a Qwen3.5 MoE variant with 35B total parameters, only 3B active per token, 256 experts, and multi-token prediction. On my RTX 3060 Ti with 8GB VRAM, it runs at 128k context with 30-36 tokens per second generation speed and 90%+ draft acceptance on optimized runs.<\/p>\n<p>That&#8217;s not a typo. A 35B model. On 8GB. At 128k context. Here are the numbers.<\/p>\n<h2>What Makes Ornith Different<\/h2>\n<p>Ornith isn&#8217;t a new architecture from scratch. It&#8217;s built on the <a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3.5-35B-A3B\" target=\"_blank\" rel=\"noopener\">Qwen3.5 MoE<\/a> base with several critical modifications:<\/p>\n<ul>\n<li><strong>35B total parameters<\/strong> across 256 experts<\/li>\n<li><strong>Only 3B active per token<\/strong> &#8211; the model picks 8 experts per token, so the compute cost is closer to a 3B model<\/li>\n<li><strong>40 transformer layers + 1 MTP layer<\/strong> for multi-token prediction<\/li>\n<li><strong>262,144 train context<\/strong> &#8211; the model was trained on massive context windows, not fine-tuned on top of them<\/li>\n<li><strong>APEX optimization<\/strong> &#8211; the &#8220;APEX-I-Mini&#8221; variant is optimized for inference efficiency<\/li>\n<\/ul>\n<p>The MoE architecture is the reason this model fits in 8GB VRAM. A 35B dense model would need ~17GB in Q4 quantization &#8211; impossible on a 3060 Ti. But Ornith only activates 3B parameters per token, so the active memory footprint is dramatically smaller. The full 35B parameters live in memory, but the compute pipeline only touches 3B at any given moment.<\/p>\n<h2>The Benchmark Results<\/h2>\n<p>I ran Ornith through multiple configurations to find the sweet spot. Every number below is from a real test run &#8211; no estimates, no projections.<\/p>\n<h3>Baseline: q8_0\/q8_0 KV Cache (No TurboQuant)<\/h3>\n<table>\n<tr>\n<td>Config<\/td>\n<td>Context<\/td>\n<td>Prompt t\/s<\/td>\n<td>Gen t\/s<\/td>\n<td>Notes<\/td>\n<\/tr>\n<tr>\n<td>UD-IQ4_NL weights, q8 KV, np=4<\/td>\n<td>32k<\/td>\n<td>~40<\/td>\n<td>~18<\/td>\n<td>Default config, moderate speed<\/td>\n<\/tr>\n<\/table>\n<p>This is what you get with standard llama.cpp settings. 18 t\/s is usable but not fast. The q8_0\/q8_0 KV cache consumes too much VRAM to allow larger context sizes.<\/p>\n<h3>TurboQuant+ Asymmetric KV: q8_0 K + turbo2 V<\/h3>\n<p>This is where the magic happens. By compressing the Value side of the KV cache to turbo2 (~2-bit) while keeping Keys at q8_0 (8-bit), the total KV cache shrinks by roughly 4x. That freed VRAM goes directly into context window size.<\/p>\n<table>\n<tr>\n<td>Config<\/td>\n<td>Context<\/td>\n<td>Prompt t\/s<\/td>\n<td>Gen t\/s<\/td>\n<td>Draft Acceptance<\/td>\n<td>Notes<\/td>\n<\/tr>\n<tr>\n<td>ncmoe 34, MTP n=2, np=1<\/td>\n<td>128k<\/td>\n<td>670-720<\/td>\n<td>30-36<\/td>\n<td>62.7% &#8211; 90.3%<\/td>\n<td>Best overall<\/td>\n<\/tr>\n<\/table>\n<p>The key optimizations beyond the KV cache:<\/p>\n<p><strong>Reduced parallel slots (np=1)<\/strong>: The default auto(4) creates 4 parallel request slots, each reserving its own KV cache. Since I&#8217;m a single user, I only need one slot. Setting np=1 freed roughly 600MB+ of VRAM that was being wasted on empty slots.<\/p>\n<p><strong>MoE layer tuning (ncmoe 34)<\/strong>: This controls how many MoE expert layers run on GPU versus CPU. At 34, the model fits perfectly. At 35, it starts spilling. At 36+, performance degrades.<\/p>\n<p><strong>MTP draft-n=2<\/strong>: Multi-token prediction drafts 2 tokens ahead and verifies them in parallel. The draft acceptance rate varies from 62% to 90% depending on the prompt complexity. At 90%, you effectively get nearly 2x the generation speed for free.<\/p>\n<h3>What the Numbers Mean in Practice<\/h3>\n<p>670-720 t\/s prompt ingestion means a 4000-token prompt (roughly 3 pages of code) processes in about 5-6 seconds. You paste a file, blink, and the model is already generating.<\/p>\n<p>30-36 t\/s generation means the model writes code about as fast as you can read it. Not instant, but never a bottleneck. Most coding tasks involve short outputs &#8211; a function, a refactor, a review comment &#8211; and at 30+ t\/s those complete in seconds.<\/p>\n<p>90% draft acceptance on optimized runs means the speculative decoder is getting it right almost every time. The model predicts the next 2 tokens, verifies them, and accepts. When it works this well, the effective speed doubles.<\/p>\n<h2>The Full Server Configuration<\/h2>\n<p>Here&#8217;s the exact command that produces these numbers:<\/p>\n<pre style=\"background:#0f172a;color:#e2e8f0;padding:1.2rem 1.5rem;border-radius:8px;overflow-x:auto;line-height:1.6;font-size:14px;font-family:monospace;white-space:pre;margin:1.5rem 0;border:1px solid #1e293b\"><code>llama-server \\\n  -m Ornith-1.0-35B-MTP-APEX-I-Mini.gguf \\\n  --spec-type draft-mtp \\\n  --spec-draft-n-max 2 \\\n  -ngl 999 \\\n  -ncmoe 34 \\\n  -fa on \\\n  -ctk q8_0 \\\n  -ctv turbo2 \\\n  -c 128300 \\\n  -b 512 \\\n  -ub 2048 \\\n  --no-mmap \\\n  -t 5 \\\n  -np 1<\/code><\/pre>\n<p>If you&#8217;re not using TurboQuant+ (the <a href=\"https:\/\/github.com\/TheTom\/llama-cpp-turboquant\" target=\"_blank\" rel=\"noopener\">llama-cpp-turboquant<\/a> fork), you won&#8217;t have the <code style=\"background:#1e293b;color:#e2e8f0;padding:2px 6px;border-radius:4px;font-size:0.9em\">turbo2<\/code> KV cache type. The closest equivalent in standard llama.cpp would be <code style=\"background:#1e293b;color:#e2e8f0;padding:2px 6px;border-radius:4px;font-size:0.9em\">q4_0<\/code> for the V side &#8211; it&#8217;s not as good, but it&#8217;s the same idea: compress Values harder than Keys.<\/p>\n<h2>How I Found the Optimal MoE Layer Count<\/h2>\n<p>The ncmoe value is the VRAM dial. Too low and you&#8217;re running half the model on CPU. Too high and you spill over 8GB. Here&#8217;s what I measured:<\/p>\n<table>\n<tr>\n<td>ncmoe<\/td>\n<td>Behavior<\/td>\n<\/tr>\n<tr>\n<td>20<\/td>\n<td>Fits easily but slow &#8211; too many layers on CPU<\/td>\n<\/tr>\n<tr>\n<td>25<\/td>\n<td>Better but not optimal<\/td>\n<\/tr>\n<tr>\n<td>30<\/td>\n<td>Good balance, solid speed<\/td>\n<\/tr>\n<tr>\n<td>34<\/td>\n<td>Sweet spot &#8211; full speed, fits in 8GB<\/td>\n<\/tr>\n<tr>\n<td>35<\/td>\n<td>Starts spilling to RAM<\/td>\n<\/tr>\n<tr>\n<td>36+<\/td>\n<td>Degraded &#8211; VRAM overflow<\/td>\n<\/tr>\n<\/table>\n<p>The difference between ncmoe 30 and ncmoe 34 is noticeable &#8211; roughly 5-8 t\/s faster generation. The difference between 34 and 36 is catastrophic &#8211; the model starts paging to system RAM and speed drops to single digits.<\/p>\n<p>The safe approach: start at ncmoe 30, verify it works, then increment by 1 until you see VRAM pressure. Every system is different &#8211; your background processes, driver version, and OS memory management all affect the exact number.<\/p>\n<h2>Draft Acceptance: Why It Varies So Much<\/h2>\n<p>One thing that surprised me during testing was how much draft acceptance varies between runs. I measured anywhere from 62% to 90%+ on the same model with the same config.<\/p>\n<p>The difference is task complexity. When the model is generating boilerplate code, repetitive structures, or simple logic, the MTP draft predictions are almost always correct &#8211; 85-90% acceptance. When it&#8217;s reasoning through a complex algorithm or making a non-obvious architectural decision, the draft predictions are less reliable &#8211; 62-65%.<\/p>\n<p>The practical impact: simple tasks run at effectively 50-60 t\/s (with 90% acceptance), while complex reasoning runs at 25-30 t\/s (with 62% acceptance). Both are usable. The model adapts its speed to the difficulty of the task.<\/p>\n<h2>Why I Chose Ornith Over Qwen3.6<\/h2>\n<p>Both models are 35B MoE with MTP. Both run on the same hardware. I tested both extensively. Here&#8217;s why Ornith won:<\/p>\n<p><strong>Stability at 128k context<\/strong>: Qwen3.6 works at 128k but with more VRAM pressure. Ornith handles 128k comfortably at ncmoe 34.<\/p>\n<p><strong>Draft acceptance consistency<\/strong>: Qwen3.6 MTP n=1 hits 74% draft acceptance &#8211; good. Ornith with MTP n=2 hits up to 90% &#8211; better.<\/p>\n<p><strong>Output quality for coding<\/strong>: This is subjective, but Ornith produces cleaner code with fewer hallucinated imports and more consistent style. It feels like it was fine-tuned specifically for the agentic coding workflow.<\/p>\n<p><strong>APEX optimization<\/strong>: The APEX-I-Mini variant has specific inference optimizations that aren&#8217;t in the base Qwen3.5 architecture. The prompt ingestion speed of 670-720 t\/s is significantly faster than what I got with any other 35B model.<\/p>\n<h2>What I Use This For Daily<\/h2>\n<p>Ornith is my &#8220;senior developer&#8221; model. I don&#8217;t use it for everything &#8211; it&#8217;s overkill for writing a simple CSS class or generating boilerplate. I reach for it when:<\/p>\n<ul>\n<li><strong>Architectural decisions<\/strong>: &#8220;I&#8217;m splitting this Laravel app into a Vue SPA with API routes. What&#8217;s the best state management approach?&#8221;<\/li>\n<li><strong>Code review<\/strong>: &#8220;Review this WooCommerce plugin for security issues and performance bottlenecks.&#8221;<\/li>\n<li><strong>Complex debugging<\/strong>: &#8220;This Go concurrency pattern is causing a race condition. Here&#8217;s the code. Find it.&#8221;<\/li>\n<li><strong>Large context analysis<\/strong>: Feed it 2000 lines of code and ask &#8220;What&#8217;s wrong with this module?&#8221;<\/li>\n<\/ul>\n<p>For quick tasks, I use Gemma 4 E4B at 83 t\/s. For daily coding, Ornith 35B-A3B at 30-36 t\/s. Ornith is the heavy artillery &#8211; and at 30-36 t\/s, it&#8217;s fast enough that I never feel like I&#8217;m waiting for the AI to catch up.<\/p>\n<h2>The Numbers Don&#8217;t Lie<\/h2>\n<p>I&#8217;ve been running this setup for weeks now. The 128k context window means I can feed it entire codebases. The 30-36 t\/s speed means I never wait. The 90% draft acceptance means speculative decoding is actually earning its keep.<\/p>\n<p>A 35B model on 8GB VRAM was supposed to be impossible. With the right model, the right quantization, and the right KV cache compression, it&#8217;s not just possible &#8211; it&#8217;s my daily driver.<\/p>\n<p>The local AI space is moving fast. New models drop every week. But Ornith 35B-A3B on an RTX 3060 Ti with TurboQuant+ is the setup that made me stop looking at cloud APIs. When your local model is this good and this fast, the subscription fees start looking like a tax on ignorance.<\/p>\n<p>For the model comparison that led to Ornith, see <a href=\"https:\/\/nassimstudio.com\/blog\/gemma-4-vs-qwen-vs-ornith-local-llm-coding-comparison\/\" target=\"_blank\" rel=\"noopener\">Gemma 4 vs Qwen vs Ornith: Which Local LLM Codes Best on 8GB<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I tested every model that claims to be good for local coding. Gemma 4, Qwen3-Coder, Qwen3.6 MTP &#8211; all of them. The model that actually earned a permanent spot on my machine is one most people haven&#8217;t heard of: Ornith-1.0-35B-MTP-APEX-I-Mini. It&#8217;s a Qwen3.5 MoE variant with 35B total parameters, only 3B active per token, 256 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1777,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"rank_math_title":"","rank_math_description":"Ornith 35B on 8GB VRAM: 128k Context at 30 t\/s. How I run a 35B MoE model at 128k context with 90% draft acceptance on consumer GPUs.","rank_math_focus_keyword":"ornith 35b 8gb vram 128k context","rank_math_canonical_url":"https:\/\/nassimstudio.com\/blog\/ornith-35b-8gb-vram-128k-context-benchmarks\/","_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[42],"tags":[17,34,32,8],"class_list":["post-1768","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-llm","tag-automation","tag-productivity","tag-tools","tag-wordpress"],"blocksy_meta":[],"jetpack_featured_media_url":"https:\/\/nassimstudio.com\/blog\/wp-content\/uploads\/2026\/07\/2_thumb.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1768","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/comments?post=1768"}],"version-history":[{"count":9,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1768\/revisions"}],"predecessor-version":[{"id":1868,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1768\/revisions\/1868"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/media\/1777"}],"wp:attachment":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/media?parent=1768"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/categories?post=1768"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/tags?post=1768"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}