FAST54 · Q5_K_S vs Q8
Same draft, runtime and benchmark settings for both quantizations. Both attempted all 54 cases.
Passes on 53 matched cases
| Component | Q5_K_S | Q8 |
|---|---|---|
| ARC | 5/18 | 7/18 |
| IFEval | 19/26 | 18/26 |
| HumanEval+ | 3/5 | 3/5 |
| Tool-Eval | 1/4 | 3/4 |
| Matched total | 28/53 | 31/53 |
| Wall time | 7m 29s | 7m 13s |
HumanEval · Q8 + DSpark7
HumanEval/0–9, one instruction-chat completion per task, graded with the EvalPlus base and extended tests.
Base and extended10/10Passed both test sets
Decode83.7 tok/sAcross the ten samples
Generation wall68.43 sSum of request times
All ten responses stopped naturally. Decode throughput and generation wall time were measured on those first responses.
HermesAgent-20 · Q8 + DSpark7
20 scenarios on Ciru with Chromium, seed 15035, temperature 0.6, top-p 0.95, top-k 20, min-p 0, thinking off and a 40-turn budget.
Mean verifier score85/100
Scenarios passed14/20
Wall time13m 20s
| Case | Score / 100 | Time | Outcome |
|---|---|---|---|
| HA-01 | 100 | 5.9 s | Pass |
| HA-02 | 100 | 7.4 s | Pass |
| HA-03 | 100 | 4.7 s | Pass |
| HA-04 | 100 | 22.5 s | Pass |
| HA-05 | 90 | 31.5 s | Partial |
| HA-06 | 100 | 35.1 s | Pass |
| HA-07 | 100 | 339.0 s | Pass |
| HA-08 | 100 | 22.5 s | Pass |
| HA-09 | 100 | 38.9 s | Pass |
| HA-10 | 50 | 60.5 s | Fail |
| HA-11 | 100 | 11.0 s | Pass |
| HA-12 | 20 | 63.9 s | Turn limit |
| HA-13 | 100 | 17.5 s | Pass |
| HA-14 | 70 | 5.6 s | Fail |
| HA-15 | 100 | 19.0 s | Pass |
| HA-16 | 100 | 9.1 s | Pass |
| HA-17 | 50 | 42.0 s | Partial |
| HA-18 | 100 | 7.0 s | Pass |
| HA-19 | 100 | 26.0 s | Pass |
| HA-20 | 20 | 5.4 s | Fail |