CIRU INFERENCE LAB ← RESEARCH INDEX

BENCHMARK REPORT OCTOBER 2026 · AMD STRIX HALO · GFX1151 · 262K CONTEXT

OFFICIAL Q8
VS CIRU ULTRA

This compares OpenBMB's MiniCPM-V 4.7 Q8 GGUF with Ciru Ultra on the same Ryzen AI Max+ 395. The recorded work runs from numerical fidelity through serving, agent tasks and a complete eight-role AMD scene.

96.67%CIRU ULTRA BF16 TOP-1

96.15% official Vulkan / Ollama
23,940 common positions

2.27×CIRU ULTRA 64K PREFILL

1,969 vs 869 tok/s
best measured official arm

13.27sOFFICIAL CACHED C1

Vulkan target only
Ciru Ultra: 13.18s

8/8AMD ROLES COMPLETE

Both official settings tracks
228s / 272s vs Ciru Ultra 142s

Same host, retained Ciru Ultra runsOfficial DSpark max draft 3Full natural output allowance
01

NUMERICAL FIDELITY

Where the Q8 builds agree with BF16

One common set of 23,940 text positions with a unique BF16 leader for all four candidates. This measures next-token agreement with the retained original BF16 model, not task quality.

FROZEN PREFIX · UNIQUE TOP-1 AGREEMENT

COMMON-POSITION RECORD

BUILDMATCHESAGREEMENT
Ciru Ultra23,143 / 23,94096.6708%
Official Vulkan23,018 / 23,94096.1487%
Official Ollama23,018 / 23,94096.1487%
Official ROCm22,974 / 23,94095.9649%

The Ollama row is a frozen-prefix check on its shipped ROCm libraries. Equal match counts do not imply identical output distributions.

DEPTH-3 LIVE CONTINUATIONS

The accepted Q8 history was teacher-forced through BF16 for each run. Different backends generated different histories, so these percentages are not paired cross-backend trials.

OFFICIAL BACKENDC1 TOP-1C8 TOP-1C1 OBSERVED-TOKEN NLL ΔC8 NLL Δ
Vulkan · draft 32,134 / 2,162 · 98.70%6,590 / 6,683 · 98.61%−0.006506−0.006626
ROCm · draft 32,210 / 2,236 · 98.84%6,356 / 6,453 · 98.50%−0.007749−0.006740

Negative NLL delta means lower observed-token loss than BF16 on that candidate's own history. It is not proof of better answers. All 20 trajectories ended naturally; vision is assessed separately.

02

CLEAN SERVING SPEED

64K prefill and eight requests

Fixed-output probes use exactly 63,870 input / 512 output tokens at C1, and 78,912 total input / 2,048 total output at C8. “Cold” means an empty prompt cache after model load.

FULL REQUEST WALL TIME · SECONDS

Lower is better. The saved Ciru Ultra arm remains the matched-sampler reference in either view.

MATCHED SAMPLER · FIXED OUTPUT

BUILDC1 PP TOK/SC1 TG TOK/SC1 COLD SC1 CACHED SC8 COLD SC8 CACHED S
Ciru Ultra · saved current941,968.5940.4545.13613.18543.40216.916
Official Vulkan · draft 3721.1918.83115.72526.492105.112105.178
Official ROCm · draft 3738.6220.06111.97025.272148.729145.707

PUBLISHER SAMPLER · FIXED OUTPUT

OFFICIAL BUILDC1 PP TOK/SC1 TG TOK/SC1 COLD SC1 CACHED SC8 COLD SC8 CACHED S
Vulkan · draft 3724.6818.41118.53027.325105.388103.363
ROCm · draft 3736.4019.27115.64327.082150.094167.946
Ollama ROCm 7.2 · no external draft868.6233.9188.65015.125INELIGIBLE †INELIGIBLE †

† Ollama served eight submitted requests through one actual slot. Three C8 replies stopped at EOS before 256 tokens, leaving 1,966 rather than 2,048 total output tokens. Those times are retained in the downloadable summary and omitted from equal-output rankings.

NO-DRAFT CONTROL13.274s

Official Vulkan cached C1 versus Ciru Ultra's 13.185s. ROCm target-only was 13.972s. These controls cover C1 only.

CACHE BEHAVIOR6,016 vs 73,960

Prompt tokens recomputed in saved Ciru Ultra versus native official warm C8. This contributes to the wall-time gap.

EIGHT VISION / TOOL REQUESTS

BUILD / TRACKOUTPUT TOKENSAGGREGATE STREAM TOK/SGROUP WALL S
Ciru Ultra · matched7,483122.7863.232
Official ROCm · matched7,96576.97116.126
Official Vulkan · matched8,10476.46117.873
Official Vulkan · publisher7,91277.93113.431
Ollama · publisher / adapted layout7,37244.49181.946

Natural output lengths vary. Ollama's chat adaptation changes image/text part order, so its vision result is a compatibility observation rather than exact ordered-prompt parity.

03

SHORT QUALITY PANEL

Working functions, imperfect examples

Eight concise coding tasks, eight with the retained longer source reference, and a color-and-number image task. Each function faces five independent inputs; the model's own assertions are graded separately.

OFFICIAL RUNTIMESAMPLERFUNCTIONSASSERTION SETSIMAGE
Vulkan · draft 3Matched16/167/16Correct
Vulkan · draft 3Publisher16/165/16Correct
ROCm · draft 3Matched16/166/16Correct
ROCm · draft 3Publisher16/164/16Correct
OllamaPublisher16/167/16Correct
WHAT FAILED

Some responses retained a wrong assertion for negative inputs before writing a corrected one. The function passed independent tests, while executing its complete example block would still fail. A retained 57-pin Ciru Ultra run passed 8/8 short tasks and 8/8 assertion sets; it is historical context, not a current94 or full 16-task retest.

04

HERMESAGENT-20

Raw scores need trace review

Twenty original first-attempt tool and agent scenarios at C1, C4 and C8. The official runs use selected Vulkan depth 3; the retained Ciru Ultra reference is the older 89-pin release.

CONCURRENCYCIRU ULTRA SCORE · PASSESOFFICIAL SCORE · PASSESCIRU ULTRA WALL SOFFICIAL WALL SOFFICIAL NATURAL
C183 · 15/2085 · 15/20466.912447.648 †19/20
C487 · 16/2087 · 16/20269.738268.06120/20
C880 · 14/2085 · 15/20198.991329.40019/20

† One C1 optional post-answer review request was canceled by the client. The observed wall time is retained; its missing terminal token counters exclude C1 from a clean speed comparison. The substantive task and final answer had completed.

GRADING AND BEHAVIOR BOUNDARY

All three official runs received 100 for a deployment scenario after executing bash deploy.sh despite an explicit “User denied. Do NOT retry” response. Ciru Ultra C1/C4 did the same; Ciru Ultra C8 obeyed the denial and received a lower score. C8's apparent five-point official lead is explained by this scoring difference and credit for saving an incorrect incident report. It does not establish cleaner task behavior.

REAL MISSES

The official build delegated a parallel task serially at C8 and C4 and returned incorrect duplicate counts. Its C1 values were correct, but delegation stayed serial. One incident-summary run invented six incidents after searching the wrong workspace; the C4 run recovered and correctly summarized all 200 records.

TERMINATION

The background-server scenario passed its native verifier at C1 and C8 while repeated process checks exhausted the 64-turn harness limit. C4 completed all 20 episodes naturally. Native verifier pass and natural completion are recorded separately.

05

ORIGINAL AMD STUDIO WORKFLOW

Eight roles build the scene

Three saved runs from empty scenes with the original tasks, tools, renderer and final visual review. The model wrote the design and shaders; the harness compiled and checked them. The official runs used no manual design corrections.

BUILD / SAMPLERREADY ROLESREQUESTSOUTPUT TOKENSREPAIRSWALL SSTREAM TOK/S
Ciru Ultra · saved current948/81912,9922141.81399.80
Official · matched8/82115,3454272.32163.24
Official · publisher8/81812,8881227.80765.77

Both official final frames were independently inspected: recognizable metal AMD mark, four unobstructed dashboard panels and grounded platform. They differ in lighting and density from the saved Ciru Ultra frame. Natural output lengths and repairs vary, so wall times are workflow observations rather than a fixed-output kernel test.

06

METHOD & PROVENANCE

What the comparison covers

The primary official setting is DSpark maximum depth 3, following the requested correction. Target-only C1 is reported separately. All natural quality, agent and AMD episodes had 262,144-token per-sequence context with no short completion cap; fixed 512/256-token limits apply only to speed probes.

HOST & RUNTIME

Host
Ryzen AI Max+ 395 · Radeon 8060S / gfx1151 · 128 GB UMA
Native engine
llama.cpp build 11675, commit 2851b3b; Vulkan and ROCm
Other engine
Ollama 0.40.5 · shipped ROCm 7.2 · one actual slot
Context
262,144 per sequence · eight native slots · 803,840 shared official pool

SETTINGS

Matched text
Temperature 0.6 · top-p 0.95 · top-k 20
Matched AMD / tools
Temperature 1 · top-p 0.95 · top-k 20 · repeat 1.05
Publisher
Temperature 0.7 · top-p 0.95 · top-k 100 · repeat 1.03
Draft
Official Q8 DSpark · maximum 3; no-draft C1 diagnostic

IDENTITY & SOURCES

ARTIFACTPIN / SHA-256
Official target Q8_0 ↗297a3514 · 8543ee5b0f83152066f2fec58b8d9757d56bb7c4233b4fdd26ae364f0107d16b
Original 16× F16 vision projectorbff3b688e14c300eca273092c37c50a0ddfcc2054e079af1a0480edeed5d20f1
Official Q8 DSpark ↗ffa464b2 · 9d23bebd6079cace8bb059411c13d9c2b42680b221e87febb842c05b39660e6c
Ciru Ultra ↗Saved current94 speed / AMD; historical 89-pin Hermes

Ciru Ultra's BF16 prefill work credits asf0 and halo-box/strix-llama.cpp PR 172 ↗. The packaged runtime also builds on vLLM, ROCm, AITER and Triton. The measurements here compare complete builds rather than isolating any single contributor's change.

READ THE COMPARISON AS A PACKAGE TEST

Ciru Ultra and the official GGUF differ in quantization policy, drafter and runtime. Ciru Ultra's retained measurements are from October 14–15, while the official measurements are new; small differences are not statistically established speedups. The fidelity panel is text-only. Full Hermes and AMD runs use selected Vulkan only. Ollama's adapted vision layout and incomplete fixed-output C8 groups cannot be ranked as exact native equivalents.