{"id":1770,"date":"2026-07-17T07:50:00","date_gmt":"2026-07-17T07:50:00","guid":{"rendered":"https:\/\/nassimstudio.com\/blog\/?p=1770"},"modified":"2026-07-22T15:25:34","modified_gmt":"2026-07-22T15:25:34","slug":"mtp-speculative-decoding-explained-speed-doubling","status":"publish","type":"post","link":"https:\/\/nassimstudio.com\/blog\/mtp-speculative-decoding-explained-speed-doubling\/","title":{"rendered":"Speculative Decoding Explained: How MTP Turns 18 t\/s Into 36 t\/s"},"content":{"rendered":"

I was stuck at 18 tokens per second on my local 35B model. Usable, but not fast. Every coding task involved waiting – paste a prompt, watch the tokens trickle out, wait for the response. Then I enabled multi-token prediction (MTP) with speculative decoding, and the speed doubled to 36 t\/s. Same model. Same GPU. Same VRAM. Just a different decoding strategy.<\/p>\n<\/p>\n

Most people have never heard of speculative decoding. It’s one of those techniques that sounds complex but has a dead-simple core idea: instead of generating one token at a time, the model guesses the next two tokens, checks if the guesses are right, and keeps the ones that are. When the guesses are right 70-90% of the time, you get nearly double the throughput for free.<\/p>\n

Here’s how it works in practice, with real numbers from my setup.<\/p>\n

How Normal Decoding Works<\/h2>\n

Standard autoregressive decoding generates one token at a time:<\/p>\n

    \n
  1. The model processes the entire context + the last generated token<\/li>\n
  2. It outputs a probability distribution over the vocabulary<\/li>\n
  3. The highest-probability token is selected (or sampled with temperature)<\/li>\n
  4. That token becomes part of the context<\/li>\n
  5. Repeat from step 1<\/li>\n<\/ol>\n

    The bottleneck is step 1. Every single token requires a full forward pass through the entire model. For a 35B model, that’s billions of multiply-accumulate operations per token. The GPU is fully utilized, but the throughput is limited by the model size.<\/p>\n

    How Speculative Decoding Changes the Math<\/h2>\n

    Speculative decoding uses a “draft” mechanism to predict multiple tokens ahead, then verifies them in a single pass:<\/p>\n

      \n
    1. The draft mechanism (MTP layer) predicts the next N tokens – these are cheap guesses<\/li>\n
    2. The full model verifies all N guesses in a single forward pass<\/li>\n
    3. Correct guesses are accepted – you get N tokens for the cost of roughly 1 verification<\/li>\n
    4. Incorrect guesses are rejected – the model outputs the correct token and restarts<\/li>\n<\/ol>\n

      The key insight: verification is cheaper than generation. Checking whether a predicted token is correct requires one forward pass that produces N token predictions. But generating those N tokens one-by-one would require N separate forward passes. So if the draft is right 80% of the time, you save roughly 80% of the compute.<\/p>\n

      Multi-Token Prediction (MTP)<\/h2>\n

      MTP is the specific draft mechanism used by models like Qwen3.6<\/a> and Ornith<\/a>. These models have a dedicated MTP layer – an extra transformer layer that predicts multiple tokens simultaneously.<\/p>\n

      The MTP layer sits on top of the main model. After the model processes a sequence, the MTP layer takes the hidden states and outputs predictions for the next 2-4 tokens. These predictions are then verified by the main model in a single pass.<\/p>\n

      In llama-cpp-turboquant<\/a>, you control this with:<\/p>\n

      --spec-type draft-mtp \\\\\n--spec-draft-n-max 2<\/code><\/pre>\n
      The draft-n-max<\/code> parameter controls how many tokens the MTP layer predicts ahead. The question is: what’s the optimal value?<\/p>\n
      My MTP Benchmarks<\/h2>\n
      I tested draft-n values from 1 to 4 on the Qwen3.6 35B-A3B MTP model at 32k context. Every config was the same model, same GPU, same quantization – only the draft-n value changed.<\/p>\n\n\n\n\n\n\n
      Draft-n<\/td>\nPrompt t\/s<\/td>\nGen t\/s<\/td>\nDraft Acceptance<\/td>\nVerdict<\/td>\n<\/tr>\n
      1<\/td>\n361.67<\/td>\n27.19<\/td>\n74.82%<\/td>\nBest 32k<\/td>\n<\/tr>\n
      2<\/td>\n335.65<\/td>\n26.24<\/td>\n62.69%<\/td>\nGood<\/td>\n<\/tr>\n
      3<\/td>\n320.05<\/td>\n~21-22<\/td>\n–<\/td>\nSlower than n=1<\/td>\n<\/tr>\n
      4<\/td>\n302.92<\/td>\n~11-12<\/td>\n–<\/td>\nCrashed \/ too slow<\/td>\n<\/tr>\n<\/table>\n
      The results were clear: draft-n=1 is the sweet spot<\/strong>.<\/p>\n
      Here’s why:<\/p>\n
      n=1<\/strong>: Predicts 1 token ahead. 74.82% acceptance. The model correctly guesses the next token nearly 3 out of 4 times. At 27.19 t\/s, this is the fastest generation speed I measured on any 35B model at 32k context.<\/p>\n
      n=2<\/strong>: Predicts 2 tokens ahead. 62.69% acceptance – lower because predicting 2 tokens is harder than predicting 1. But when both are accepted, you get 2 tokens for the price of 1 verification. The math works out to 26.24 t\/s – slightly slower than n=1 because the verification overhead is higher.<\/p>\n
      n=3<\/strong>: Predicts 3 tokens ahead. The draft acceptance drops further, and the verification overhead starts outweighing the gains. At ~21-22 t\/s, it’s slower than not using speculative decoding at all.<\/p>\n
      n=4<\/strong>: The draft predictions are too aggressive. The overhead of generating and verifying 4 tokens ahead kills the pipeline. At ~11-12 t\/s, it’s half the speed of n=1.<\/p>\n
      Why n=1 Wins<\/h2>\n
      The optimal draft-n depends on the draft acceptance rate. Higher acceptance = more value from higher n. Lower acceptance = more overhead from failed drafts.<\/p>\n
      At 74.82% acceptance (n=1), each accepted draft saves roughly one full forward pass. At 62.69% acceptance (n=2), the second token is wrong 37% of the time, which means the verification often only accepts 1 of the 2 drafted tokens – but you still paid the overhead of drafting and verifying 2.<\/p>\n
      The breakeven point depends on the ratio of draft cost to verification cost. For a 35B MoE model where verification is cheap (only 3B parameters active per token), the breakeven is roughly at 70% acceptance for n=2. Below that, n=1 is faster.<\/p>\n
      The Ornith model hits 90%+ draft acceptance on simple tasks (boilerplate, repetitive code, structured output). On complex reasoning, it drops to 62-65%. The average across all tasks is roughly 70-80%, which is right at the n=1\/n=2 boundary.<\/p>\n
      This is why I settled on --spec-draft-n-max 2<\/code> as my config – it allows n=2 drafts on simple tasks where acceptance is high, while falling back to n=1 on complex tasks where the second draft would be rejected anyway.<\/p>\n
      How to Enable MTP on Your Setup<\/h2>\n
      If your model supports MTP (check the model card on HuggingFace – it will say “MTP” or “multi-token prediction”), enable it with:<\/p>\n
      llama-server \\\\\n  -m model.gguf \\\\\n  --spec-type draft-mtp \\\\\n  --spec-draft-n-max 2 \\\\\n  -ngl 999 \\\\\n  -ncmoe 34 \\\\\n  -fa on \\\\\n  -ctk q8_0 \\\\\n  -ctv turbo2 \\\\\n  -c 128300 \\\\\n  -np 1<\/code><\/pre>\n
      The --spec-type draft-mtp<\/code> flag activates the MTP layer for speculative decoding. Without it, the model runs standard autoregressive decoding at whatever base speed it can achieve.<\/p>\n
      Not all models support MTP. Standard Qwen2.5 models, Gemma 4, and most older models don’t have an MTP layer. Check before enabling – the flag will be silently ignored if the model doesn’t support it.<\/p>\n
      The Real-World Impact<\/h2>\n
      Going from 18 t\/s to 30-36 t\/s (with MTP + turbo2 KV cache combined) changed how I use local AI. At 18 t\/s, every interaction involved waiting. I’d paste a prompt and watch the output stream in, checking my phone while it generated.<\/p>\n
      At 30-36 t\/s, the output arrives as fast as I can read it. For short coding tasks – generate a function, write a test, refactor a block – the response completes in 2-3 seconds. For longer tasks – code review, architectural analysis – the model generates at the speed of thought.<\/p>\n
      The combination of MTP speculative decoding and asymmetric KV cache compression is what makes 8GB VRAM viable for serious local AI. Neither optimization alone is enough. MTP doubles the speed, and the KV cache compression provides the VRAM headroom for large context. Together, they turn a mid-range GPU into a serious coding assistant.<\/p>\n
      The local AI space loves talking about model size and parameter count. But the real performance gains come from infrastructure optimizations like MTP and KV compression – the stuff that happens around the model, not inside it. If you’re not using speculative decoding on a model that supports it, you’re leaving half your GPU’s performance on the table.<\/p>\n
      For the benchmark numbers comparing each compression tier, see turbo2 vs turbo3 vs turbo4: Which KV Cache Quantization Fits Your VRAM<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
      I was stuck at 18 tokens per second on my local 35B model. Usable, but not fast. Every coding task involved waiting – paste a prompt, watch the tokens trickle out, wait for the response. Then I enabled multi-token prediction (MTP) with speculative decoding, and the speed doubled to 36 t\/s. Same model. Same GPU. […]<\/p>\n","protected":false},"author":1,"featured_media":1779,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"rank_math_title":"","rank_math_description":"Learn why compressing Values more than Keys unlocks 128k context on 8GB VRAM. Asymmetric KV cache compression reduces memory by 4x while preserving quality.","rank_math_focus_keyword":"asymmetric kv cache compression","rank_math_canonical_url":"https:\/\/nassimstudio.com\/blog\/asymmetric-kv-cache-compression-128k-consumer-gpu\/","_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[42],"tags":[34,32,8],"class_list":["post-1770","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-llm","tag-productivity","tag-tools","tag-wordpress"],"blocksy_meta":[],"jetpack_featured_media_url":"https:\/\/nassimstudio.com\/blog\/wp-content\/uploads\/2026\/07\/4_thumb.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1770","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/comments?post=1770"}],"version-history":[{"count":13,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1770\/revisions"}],"predecessor-version":[{"id":1915,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/posts\/1770\/revisions\/1915"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/media\/1779"}],"wp:attachment":[{"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/media?parent=1770"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/categories?post=1770"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nassimstudio.com\/blog\/wp-json\/wp\/v2\/tags?post=1770"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}