Speed Is a Craft Choice: Decoding Belongs in the Design Phase

Speed is often treated as a property a model simply has, printed beside a benchmark and accepted as fate. A recent set of results on a desktop machine tells a different story: change the decoding strategy, keep the hardware and the model, and generation moves by half again.

The numbers come with careful limits. Every figure in the table is vendor-reported, measured by NVIDIA on NVIDIA hardware under a specific test setup. Vendor figures deserve that flag every time those figures are quoted, and the flag does not make the underlying lesson disappear.

The lesson concerns design. Decoding strategy, draft length and verification behaviour now sit beside quantisation, engine builds and context budgets as decisions a team makes on purpose, measures honestly and defends with evidence from real workloads.

01 · The numbers on the table

One table in NVIDIA's October 30, 2026 technical post deserves more attention than the headline around it. Running Nemotron-3.5 Lightning on a DGX Spark in the 64GB configuration, through TensorRT Edge-LLM, NVIDIA reports mean generation speed on SpeedBench 1K rising from 73.3 tokens per second with DSpark speculative decoding off to 108.5 with DSpark on. On HumanEval prompts, the same switch moves generation from 73.4 to 126.9 tokens per second. Median time to first token barely changes — 201 milliseconds off, 203 on, for the 1K workload. Every figure is vendor-reported, measured by NVIDIA on NVIDIA hardware, under a specific test setup: batch size one, a single streaming request, a 128-token output limit, three draft tokens in the DSpark configuration.

Those test conditions are part of the result, not fine print beneath the result. Batch size one and a single streaming request describe a personal, interactive workload — exactly the setting a desktop machine serves — rather than a packed data centre queue. A team running large batches, long outputs or shared serving should expect different numbers and should measure on the workload actually served instead of borrowing a table built for another shape of work.

02 · How speculative decoding buys speed

The mechanism is a wager with a referee. A fast, cheap drafting process proposes several tokens ahead; the full model verifies the proposals in one pass, accepting the stretches that hold and correcting the first point of divergence. When drafts are good, several tokens land for roughly the cost of one full step. When drafts are poor, verification still guarantees the output matches what the full model would have produced — the speed is opportunistic, the correctness is not. Draft length, draft source and acceptance behaviour become tuning surfaces, closer to engine building than to prompting.

Tuning those surfaces is real craft work. Longer drafts promise more tokens per verification pass and waste more work when an early proposal misses. Shorter drafts stay safer and leave speed unclaimed. The right setting depends on how predictable the workload is: structured, repetitive generation drafts well, while open-ended generation argues for restraint. No single setting deserves to be called best without naming the workload the setting was tuned for.

Generation speed stops being a property a model simply has, and becomes a property a team designs, measures and defends.

03 · Why the first token barely moves

The flat time-to-first-token line is the honest detail in the table, and the instructive one. Speculative decoding accelerates the long middle of a generation; the wait for the first token is governed by prompt processing, which drafting does not touch. Two different latencies, two different levers. Systems that feel fast attend to both: prompt-side work — context size, caching, prefill efficiency — for the opening moment, and decoding strategy for everything after. Conflating the two produces optimisation programmes that improve the number nobody was waiting on.

Users feel the distinction even when dashboards hide the distinction. A long pause before the first word reads as a system that has stalled, however fast the words arrive afterwards. A quick first word followed by a slow stream reads as a system working steadily. Both experiences come from the same generation, split across two latencies, and a serious performance review reports the pair separately rather than hiding both problems inside one average.

04 · The design-phase argument

Agent workloads multiply whatever the decoding layer decides. A single user request fans out into planning, tool calls, verification and revision — dozens of generations, each paying the per-token rate. A gain near fifty percent, of the size NVIDIA reports on the 1K workload, compounds across every loop the agent runs, and lands directly in operating cost and in how responsive the system feels. That is why decoding strategy belongs in the design phase, beside quantisation, engine builds and context budgets — the cluster of decisions the discipline behind NVIDIA-Certified Professional: AI Operations treats as the craft itself, and the same cluster context engineering at Ovia weighs before an agent ever meets a user.

Design-phase attention also protects honesty later. When decoding is chosen early, the choice gets benchmarked against the intended workload, recorded beside the engine build, and revisited when hardware or models change. When decoding is left as a default discovered after launch, speed problems arrive as surprises and fixes arrive as folklore. Craft, in operations, mostly means making the important choices while evidence can still change the plan cheaply.

05 · What to watch

Watch for independent reproductions on real agent workloads rather than benchmark prompts, for drafting strategies tuned per workload instead of per demo, and for serving stacks that expose decoding choices as first-class configuration. The deeper shift is cultural: generation speed stops being a property a model simply has, and becomes a property a team designs, measures and defends — a craft choice, made on purpose.

A second discipline follows from the flat first-token line: measure the two latencies separately, in production, per workload. Benchmark tables report medians on clean prompts; real estates serve long contexts, warm and cold caches, and users who judge the whole system by the pause before the first word appears. Teams that instrument both numbers can spend effort where the waiting actually happens — prefill and context handling for the opening moment, decoding strategy for the long generation that follows — instead of celebrating an average that hides both problems at once.

The standard to hold is simple. Name the workload, publish the conditions, separate the two latencies, flag vendor figures as vendor figures, and tune the decoding layer deliberately.