Qwen3.8-27B on one RTX 3090

* atomic-agent does not self-report a version. v0.2.1 is inferred from the latest GitHub release of AtomicBot-ai/atomic-agent (2026-08-14) at install time (2026-08-16).

The architecture is what makes everything else possible

The GGUF reports general.architecture = qwen35, and it is a hybrid SSM + attention model, not a plain transformer. Only one layer in four keeps a KV cache. That single fact is why a 27B model gets a 131K window on a 24 GB card.

block_count              65        full_attention_interval  4     ← hybrid
attention.head_count     24        attention.head_count_kv  4
attention.key_length    256        attention.value_length 256
embedding_length       5120        context_length      262144
nextn_predict_layers      1        ← MTP heads ship with the model
general.sampling      temp 1.0 · top_k 20 · top_p 0.95

Most "which model / which setting" advice in circulation is eyeballed — side-by-side screenshots judged by taste — or copied from someone running different hardware. This benchmark only asks questions with a verifiable answer:

• Reasoning problems: every answer computed by a reference implementation before the model saw the question. Several sets are validated against exhaustive brute force.
• Coding tasks: scored by hidden test suites the model never sees, each suite first validated against a reference solution.
• Performance: taken at thermal steady state after discarding warm-up runs, never cold.
• Tool calling: scored on selecting the right tool, extracting the right arguments, and — critically — declining to call when no tool applies.

What is deliberately not measured: anything judged by taste. Web-design quality, prose style, "which output looks nicer". Those matter, but a benchmark that scores them is reporting the author's preferences with extra steps.

Difficulty was escalated because the model kept passing

The first problem sets were too easy — everything scored 100%, which measures the test, not the model. Difficulty was raised three times. The progression is itself a result:

Instrumentation scope

Telemetry depth is not uniform. A single long-lived nvidia-smi process sampled temperature, power, clocks and utilisation every 2 s with native timestamps, at 0.0% measured CPU overhead — no per-sample process spawning, and nothing called inside the timed path. Later probes record start/end timestamps per row and are joined against that log offline. Earlier probes carry wall-clock and token counts only.

Any request containing an image killed the server instantly. Text-only requests ran indefinitely.

ggml-cuda.cu:106: CUDA error
CUDA error: the requested functionality is not supported
  current device: 0, in function ggml_cuda_mul_mat_cublas_impl at ...:1548
  cublasGemmEx(..., cu_data_type_a, ..., cu_data_type_b, ..., cu_compute_type,
               CUBLAS_GEMM_DEFAULT_TENSOR_OP)

What it is not

• Not the mmproj dtype. The repo ships BF16 and F16 projectors. Both abort identically.
• Not memory. Same abort at -c 16384 and -c 131072, ~2 GB free at abort. The status is NOT_SUPPORTED, not OOM.
• Not image size. Aborts with and without --image-min-tokens, with --image-max-tokens 512, at 640×480 and 1920×1080.
• Not the installer. A third-party report in the upstream issue hit the same abort with a nixpkgs build. Reinstalling llama.cpp another way does not fix it.

The fix

export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32

fp16 does not help; only fp32 does. Measured cost: none. 66.11 tok/s with it against 65.28 without, same seed, thermally equalised. Prefill over a 128,290-token prompt was unchanged at 705 tok/s.

The CPU fallback (--no-mmproj-offload) also avoids the abort and is unusable: 28.6 s at 640px — and the model hallucinated the contents, because the text is illegible at that scale — rising to over 300 s at 1080p, against 3.4 s on GPU.

This is a workaround, not a fix. It forces fp32 for every cuBLAS matmul, not just the one that breaks. Upstream: ggml-org/llama.cpp#24999.

With q4_0 K+V the measured cost is 28,672 bytes per token:

VRAM_llama = weights + 28,672 B × context + 185 MiB
             (+ 1,125 MiB when the vision projector sits on the GPU)

That reproduces six independent boot measurements to within 25 MiB.

Ceilings were verified by filling them: a 128,290-token prompt with a needle hidden at the start, recovered verbatim. 262,144 OOMs. 163,840 boots and runs with ~1 GiB free.

Allocating context is nearly free. Filling it is not.

A widely-shared claim says lowering -c buys speed. Tested directly with a short prompt at every setting, which isolates allocation from fill:

Keep the window. The reported speedups from shrinking -c come from processing less context, not from allocating less. Filling it is what costs: the same prompt at 53,054 tokens of context made the model think 63% more and take nearly twice as long — while staying 4/4 correct. Accuracy holds at a full context; latency does not.

The model ships MTP heads, so --spec-type draft-mtp works with no extra download. It is off by default, and the wrong spelling fails loudly — --spec-type mtp refuses to start with unknown speculative type, so nobody is silently running without it because of a typo. They are running without it because they never set the flag.

n=2 wins on this GPU. A widely-cited figure puts the sweet spot at n=3 — that measurement was taken on an RTX 6000. The optimum depends on spare verification compute, which is a property of the card. MTP itself is the big win: +77%. Its VRAM cost is 834 MiB, not the 2–3 GB sometimes quoted.

Temperature is a speed knob, not a quality knob

The last column is the control: without MTP, temperature does nothing (38.9 → 39.3 → 38.5, flat within noise). That localises the entire effect in the speculative path. At temp 1.0 acceptance drops to 0.741 and throughput follows, ~9% down from 0.6.

The model card recommends temp 1.0. On this hardware 0.6–0.7 gives ~10% more throughput, and accuracy was identical (8/10 at both) on verifiable problems, and on code generation (99% at both 0.6 and 1.0). Scope: single-answer arithmetic and code. Temperature very plausibly matters for open-ended work, which this benchmark does not measure.

--spec-draft-p-min did not reproduce. A discussion reports 15% faster. Sweeping 0.00 → 0.50 at two temperatures moved throughput by at most 2.4–3.3%, inside or barely outside noise, with acceptance unchanged.

Two Q4 GGUFs of the same model from two publishers who each claim their dynamic quantisation is better. Perplexity on identical corpora, 60 chunks, -c 4096, f16 KV.

On general text the gap is 17× smaller than the error bar; on code, 13× smaller. In quality terms these are the same model.

What is real is the 765 MiB difference, worth about 28,000 tokens of context on a 24 GB card — the whole reason the usable window goes from 98K to 131K. AtomicChat wins on size, not on speed: thermally equalised it is 1.8% slower.

Conventional wisdom says LLM decode is memory-bandwidth bound. Not here: memory clock stayed pinned at 9501 MHz across every power level, so the only variable was compute — and throughput tracked it.

Fitting throughput against SM clock gives an exponent of 0.75 — 1.0 would be purely compute-bound, 0.0 purely memory-bound. The hybrid SSM architecture and MTP verifying three tokens per step both push it that way.

79 °C sustained is well inside spec: this card throttles at 95 °C, its maximum operating temperature is 93 °C, and the thermal-violation counter stayed at zero. Dropping to 300 W buys 74 °C for a 9.1% throughput loss — not worth it unless noise matters.

Undervolting is blocked by Wayland

The real Linux undervolt is a positive clock offset, which shifts the V/F curve so the card runs faster at the same voltage. It needs nvidia-settings with Coolbits and a running X server. Under Wayland, nvidia-settings reads values but refuses writes, and as root it cannot reach the display at all. Nothing available without X substitutes: -lgc only caps clocks, --set-vf-derate applies a negative offset, and forcing memory to its maximum with -lmc 9751,9751 reports success and does nothing — the real clock stays at 9501 and throughput does not move.

PowerMizer: leave it on Normal. Under load the GPU already sits at its highest performance level; "Prefer Maximum Performance" only pins high clocks at idle. The card does run at Performance Level 3 (memory 19002 MT/s) when Level 4 (19502) exists, but at 75% compute-bound that 2.6% of bandwidth is worth about 0.6% of throughput.

Three separate mechanisms get conflated in most discussions. They compose, and two of them are invisible unless you read the chat template.

1 · reasoning_effort — a prompt instruction, and it is silently ignored via the standard field

The template accepts exactly three values and injects a different system instruction for each:

medium is the worst option, and the template explains why: it is a valid value that passes validation and then sets no guidance. Not "medium effort" — no guidance, which scores below either explicit instruction.

2 · reasoning-budget — a hard cap, and a cap that is too low is worse than no reasoning

Advice in circulation puts the optimum at 6K–12K. Testing budgets against a fixed problem set cannot validate that: once a budget is large enough to let the model finish, every larger budget produces byte-identical output. The informative experiment is the opposite — hold the budget fixed and scale the problem until the model needs more than the cap allows.

The break is at 60 steps — exactly where natural thinking (2,954) first crosses the cap. Below it, capped and uncapped runs are identical. Above it the capped model gives up and returns nonsense: 18, 18 and 35 against true answers of 295, 2,666 and 1,932,916. Totals: no cap 4/5, reasoning off 2/5, cap 2048 2/5. An insufficient budget is as bad as no reasoning at all.

3 · The model terminates on its own

Left completely unbounded (--reasoning-budget -1, max_tokens 100000) it thought 25,772 tokens, produced a complete 4,454-character answer, and closed with finish_reason: stop after 552 s. It does not loop.

How much budget is actually useful

Thinking and the answer share one context window — there is no separate buffer. The KV cache holds prompt + thinking + answer as one sequence, and the answer is conditioned on the thinking by attending to it. max_tokens covers both. Only across turns can thinking be dropped, which is what preserve_thinking controls.

The model self-limits around 26,000 tokens. A 100K budget would be harmless — never reached — but pointless, and the arithmetic is worth stating plainly: 100,000 thinking tokens at 60 tok/s is 28 minutes for one answer, and would consume 100K of a 131K window, leaving 31K for prompt plus answer. The useful range is 26K–32K: high enough never to truncate, below the point where the model starts to wander.

Context envelope at -c 131072: a 2K prompt leaves 129K for thinking plus answer; a 50K prompt leaves 81K; a 90K prompt leaves 41K. Below about 90K of prompt, thinking never competes seriously with the answer.

Context shift is disabled by default. When the window fills, generation stops — it does not slide. Several "no answer" results in this benchmark are that ceiling biting, not the model giving up.

preserve_thinking — the vendor claim holds, with margin

Twelve-turn agentic session where every turn required real reasoning:

Preserving is 40% faster and generates 41% fewer tokens, for 69% more context. The mechanism is visible turn by turn: with preservation, thinking collapses to 25–47 tokens in the middle turns because the model can see its own earlier reasoning; without it, thinking stays at 150–470 as it re-derives each time. At turn 12 the cost is 2,227 tokens, projecting to ~7,400 at 40 turns — 5.6% of a 131K window. Worth it.

Five tasks of escalating difficulty, generated directly by the model with no harness in the loop, scored by hidden test suites. Each suite was validated against a reference implementation first; c2 and c4 additionally check random instances against exhaustive brute force.

Reasoning helps exactly at the edge of capability, and nowhere else. Below that edge it is pure cost. Above it — no case found. The price for the last 2–3% is 10–15× the wall time.

A detail worth noting: on c4, xhigh produced 48–54 lines where reasoning-off produced 86–127. Thinking led it to the compact correct algorithm rather than a longer construction.

The pathology

xhigh on c1 is actively destructive. Across nine attempts at three temperatures, six produced no usable code — the model thought 29,000–43,000 tokens for up to 16 minutes and never emitted a code block. The same task with reasoning off scores 26/26 in 11 seconds. Temperature was not the variable: 0.3, 0.6 and 1.0 all showed it. This is the single strongest argument for keeping reasoning off in a day-to-day coding profile.

Tool calling is not a weak point

An easy sweep (unambiguous requests, 5/12/19/30 tools available) scored perfect at every level, including 30 tools — zero wrong picks, zero malformed arguments. So a hard set was built around real failure modes:

12/12. This refutes a claim made earlier in this same investigation — that pruning a harness's toolsets would reduce tool-selection errors. It does not. Pruning is still worth doing, but only to reclaim context.

Three harnesses, one local model

All three passed everything, including the hard task's reversal semantics. Neither task discriminated on correctness — the outputs are genuinely independent (10–24% textual similarity, i.e. shared boilerplate). Fixed context cost is what separates them: 4.1%, 5.7% and 10.7% of a 131K window burned before any work begins, and the Hermes figure is already after pruning six toolsets, which recovered 7,973 tokens per session.

Two things had to be fixed before opencode could be measured at all. It shipped with no configuration pointing at a local model — its auth.json listed only a cloud provider — and its database did not migrate across the 1.2.18 → 1.18.18 upgrade, failing every run with no such column: name.

Across every set, the failures cluster into two kinds.

The pattern: it fails when it must hold a constraint while optimising, and when exact arithmetic runs very deep. More thinking budget does not fix either — the recurrence failed with 16,819 thinking tokens and a clean self-termination.

When it fails, it fails loudly. Of ten wrong answers collected across the session, nine were off by 25% or more — typically 39% to 99%. Only one was a near miss. That is a practically useful property: a plausible-looking number is probably right, and a wrong one usually looks absurd. Scope: numeric problems. Failure modes in prose or code may differ entirely.

A frontier model on the same problems, without tools

A script generated five fresh problems with a clock seed, writing the questions to one file and the answers to another that was not read. Opus 5 max answered them by reasoning alone, with no code execution, and committed its answers to a hashed file before any verification existed. The local model then received the same five with an explicit instruction to solve them mentally.

Read this correctly. Five for five on both sides means neither model failed on this set — a statement about the set's difficulty, not about model equivalence. Three asymmetries remain even with the commit-then-verify protocol: the problem types were chosen by Opus 5 max, the sample is five, and the hardware is not comparable. A tie proves nothing except that these five problems do not separate them.

And a broader benchmark does separate them. Artificial Analysis's Intelligence Index v4.1.1, which aggregates nine evaluations, places Qwen3.8-27B at 52 against Claude Opus 5 (max) at 63 — level with GPT-5.6 Luna, eleven points below the top. That is a third-party figure this benchmark did not verify, but it is the right corrective: a five-problem tie measures the problems, not the models. What the local model demonstrably is, on this hardware, is strong enough that finding its ceiling took three escalations of difficulty. What it is not is frontier-equivalent.

What this model is on this hardware

A 27B model with a 131,072-token window and vision at 64 tok/s sustained on a consumer card is not what a 27B usually buys. The reason is architectural: only about one layer in four keeps a KV cache, so context costs 28,672 B/token instead of the ~90 KB a dense 27B would need. Long context here is a property of the architecture, not of tuning.

Capability was rarely the limit. It solved every coding task, every tool-calling trap, and matched a frontier model on a five-problem set. It broke only on very deep exact arithmetic and on holding a constraint while optimising.

The binding constraint is VRAM

Every meaningful decision was a VRAM trade, not a throughput trade. q8_0 KV would cut the window from 131K to ~52K. The vision projector costs 1,125 MiB. The 765 MiB between two quantisations of the same model is worth 28,000 tokens. Meanwhile the desktop itself moves 500 MiB between an idle session and a working one. Speed was never the problem: a complete working module in 27 seconds is not a bottleneck.

Settings that matter, in order

1. GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32 — without it, vision is unusable.
2. --spec-type draft-mtp --spec-draft-n-max 2 — +77% for 834 MiB.
3. q4_0 KV, -np 1 — q8_0 halves the window; more slots disable MTP entirely.
4. Reasoning off for coding — same correctness, 5–85× faster, and xhigh destroyed 6 of 9 attempts at one task.
5. Temperature 0.6–0.7 — ~10% more throughput than the recommended 1.0, no measured accuracy cost on verifiable work.
6. Keep -c high — allocation is nearly free; only filling costs.

Advice that did not survive measurement

What a 24 GB card cannot do

The 1M-token windows advertised for this class of model are not a configuration away. At the measured 28,672 B/token, one million tokens needs 26.7 GiB of KV alone — more than the whole card, before weights. Even the native 262,144 does not fit (7.0 GiB of KV against 6.1 GiB available). Pushing KV to system RAM with -nkvo does not rescue it: 26.7 GiB against 31 GiB of RAM would not fit alongside the OS, and if it did, reading that KV per generated token over PCIe 4.0 x16 lands at 0.75–0.94 tok/s. Those windows run on 80 GB cards. On a 3090 it is arithmetic, not tuning.

Also out of reach: the FP8 and NVFP4 builds of this exact model, which are published and require Ada or Blackwell; and undervolting, which needs an X server.

What this benchmark does not establish

Single machine, and most configurations measured once rather than repeated — the thermally-equalised comparisons and the multi-attempt coding runs are the exceptions. Perplexity is a proxy for quantisation damage, not a functional quality benchmark. The harness comparison rests on context cost and one observation about algorithm choice, because neither coding task discriminated on correctness. Reasoning and temperature findings cover verifiable single-answer problems and code generation only — for open-ended work such as prose or visual design, where many outputs are valid, this benchmark says nothing, and temperature very plausibly behaves differently there.

Kept deliberately. Eight of these produced a stated conclusion that later measurement overturned.

• "AtomicChat is 4.5% faster." A cold run against a warm one. Equalised at 79 °C it is 1.8% slower.
• "Raising the power limit to 380 W made it slower." Same thermal confound. At matched temperature 380 W is fastest.
• "Unrestricted reasoning never answers." It hit the benchmark's own max_tokens. It terminates at ~25,772 thinking tokens.
• "Fewer tools means fewer tool-calling errors." Refuted: 12/12 on adversarial cases.
• "Reasoning never helps for coding." True for c1–c3, false for c4 and c5. Only visible once the tasks were hard enough.
• The first reasoning A/B gave reasoning-off 400 max_tokens while giving budgeted runs thousands. Seven of eight failures were truncation. Re-run with a common ceiling: 8/8.
• Testing ever-larger budgets against a fixed problem set is tautological once the smallest succeeds. Replaced by the complexity ladder.
• An acceptance-rate table was parsed with a regex that matched the wrong number, producing values that rose with temperature while speed fell. Re-measured from raw accepted/generated counts.
• Hand-derived answer keys were wrong four separate times — six of eight in the first set, three of five turns in a multi-turn probe, and twice more in coding-task expectations. In every case a capable model "failing" an easy item meant the key was wrong. Every answer here is now computed by a reference implementation.

export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32   # or every image request aborts

llama serve -m Qwen3.8-27B-AD-Q4_K_M.gguf \
  --mmproj mmproj-F16.gguf \
  -ngl 999 -fa on --jinja -np 1 \
  -c 131072 -ub 512 -b 4096 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --reasoning off \
  --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --host 0.0.0.0 --port 8080

• -np 1 — more slots disable MTP.
• q4_0 KV — q8_0 costs 57,344 B/token and drops the window to about 52K.
• -ub 512 — 1024 gives the same prefill and costs 404 MiB.
• --reasoning off — for coding. Switch on with an unbounded budget only for hard logic.
• To set reasoning effort, it must go in chat_template_kwargs, not the standard field.

The full suite — problems, reference implementations, hidden test suites, probe scripts and raw logs — is designed to be run against any other model and to be broken. A benchmark nobody can falsify is worth nothing.