Ciru Inference Labllm.ciru.ai / research

Crown Citadel Research Report

Nanbeige4.2-3B benchmarks

With DSpark7 enabled on both profiles, Q8 passed three more FAST54 cases and finished 16 seconds sooner than Q5_K_S.

FAST54 · Q5_K_S vs Q8

Same draft, runtime and benchmark settings for both quantizations. Both attempted all 54 cases.

Passes on 53 matched cases

Q5_K_S28 / 53
Q831 / 53
ComponentQ5_K_SQ8
ARC5/187/18
IFEval19/2618/26
HumanEval+3/53/5
Tool-Eval1/43/4
Matched total28/5331/53
Wall time7m 29s7m 13s

HumanEval · Q8 + DSpark7

HumanEval/0–9, one instruction-chat completion per task, graded with the EvalPlus base and extended tests.

Base and extended10/10Passed both test sets
Decode83.7 tok/sAcross the ten samples
Generation wall68.43 sSum of request times

All ten responses stopped naturally. Decode throughput and generation wall time were measured on those first responses.

HermesAgent-20 · Q8 + DSpark7

20 scenarios on Ciru with Chromium, seed 15035, temperature 0.6, top-p 0.95, top-k 20, min-p 0, thinking off and a 40-turn budget.

Mean verifier score85/100
Scenarios passed14/20
Wall time13m 20s
CaseScore / 100TimeOutcome
HA-011005.9 sPass
HA-021007.4 sPass
HA-031004.7 sPass
HA-0410022.5 sPass
HA-059031.5 sPartial
HA-0610035.1 sPass
HA-07100339.0 sPass
HA-0810022.5 sPass
HA-0910038.9 sPass
HA-105060.5 sFail
HA-1110011.0 sPass
HA-122063.9 sTurn limit
HA-1310017.5 sPass
HA-14705.6 sFail
HA-1510019.0 sPass
HA-161009.1 sPass
HA-175042.0 sPartial
HA-181007.0 sPass
HA-1910026.0 sPass
HA-20205.4 sFail