Qwen 3.5 Digit Recall

Verdicts

The 9B has 1.7 bits more to spare

The same plain ladder, built the same way, on both models. The cliff is the highest-bit build whose π recall falls below half the reference: Q5_K_M (5.9 bits per weight) on the 4B, Q3_K_M (4.1) on the 9B.

• The lead shrinks as bits drop. At 8–6 bits the 9B has 10–19× less noise on π. Its noise then grows faster (12× per bit against 7.6×), and both reach the same level at Q2_K. Across unsloth's files the gap goes from 15–19× at Q8_0–Q5_K_M to 1.2–1.5× at the 2–3 bit files.
• Per byte, the comparison reverses. At about 4.5 GB, the 4B at Q8_0 recites 1,000 digits and the 9B at Q3_K_M recites 129. The 9B matches the 4B only from 6.5 GB (Q5_K_M). Exact recall needs many bits per weight more than it needs many weights.
• The 9B's BF16 recites π past the 1,200-digit generation cap, so its reference is a lower bound. Q8_0 reaches the cap too.

The 9B: no knife-edges, sturdier copying

19 builds: unsloth's BF16 reference and 12 published files, our plain ladder from Q6_K to Q2_K, and Q2_K with our imatrix.

• Margins. The thinnest BF16 margin on π is 1.6 nats, against 0.1 on the 4B, and margins rise with depth into π (6.6 nats over the first 100 digits, 11.2 after 300). Recall still breaks at thin margins: unsloth's Q4_K_M, UD-Q4_K_XL and Q4_0 all first flip digit 722 (6th percentile of margin).
• Copying. Every build from 3.4 bits per weight up copies at 0.03–0.07 errors per 100 digits, except plain Q2_K (34). The E4b looping the 4B showed at 3.6–4.3 bpw is gone; only UD-IQ2_XXS loops.
• Prompt vs memory. 9B BF16 copies all 8 planted digits faithfully, and so does every build that can copy at all. There is no memory pull for quantization to change.
• e and √2. Both models know exactly 192 digits of e from the plain phrasing; the 9B reads "e to 1000 decimal places: 2." as a numbered list and "e = 2." as Python code.

VRAM sits below file size + 770 MiB on the 9B. Its token-embedding table is separate from the output layer, and llama.cpp keeps it in host memory. Q4_K_M fits an 8 GB card with an 8k context. Decode flattens below about 5 bits, as on the 4B, at 1.5–1.7× slower. The full 9B write-up is in REPORT_9B.md.

Recall declines early and steadily

The sections below are the 4B in detail. BF16 recites 1,000 correct digits of π (and stops there because the prompt asks for 1,000), 192 of e and 105 of √2. π has the runway, so it carries H1. Every stock quant below Q8_0 loses digits, and there is no plateau down to 4 bits: recall falls below half the reference at Q5_K_M (5.9 bits per weight).

Memorized digits are far more fragile than ordinary digit prediction. At Q4_K_M the noise on π is 12× the noise on random digits, and it grows 7.6× per bit removed against 3.2× for random digits. Bits per weight are measured from each file; they run 0.3–0.4 above the usual labels because this model's large vocabulary layers stay at higher precision.

Errors start at a few knife-edge digits

Feeding the true digits and reading the logits gives each position a margin: how far the right digit leads its best competitor. BF16 gets all 1,000 positions right with a median margin of 9.2 nats, but a handful of digits are nearly tied. Those are where recall breaks first.

• Of 77 π recalls that stopped early, the most common stops are positions 326 (8 runs), 88 (7), and 821, 10, 87 and 123 (4 each). The median stop sits at the 12th percentile of BF16's margins.
• At high bits, errors appear only at the thinnest margins: Q6_K's two teacher-forced errors are at 813 and 821. At Q4_K_M the 14 errors sit at the 7.6th percentile of margin. From Q3_K_M down, the noise exceeds most margins and errors spread everywhere.
• This also explains why recall is so uneven between neighbouring quants: our Q6_K recites 813 digits and unsloth's Q6_K 326, with almost the same noise. Both sit on the edge at digit 326 and pass or fail on it.

Copying holds; its failures are skips inside runs

Verbatim copy (E3). 40 random strings of 100 to 1,000 digits. BF16 makes 3 errors in 17,000 digits. Every build at 4.3 bits per weight or more copies 1,000-digit strings perfectly. Only three builds break, and the same quant types survive when built differently:

Spelling as words (E4b). Even BF16 drops digits inside runs: run-heavy strings get 2.2 errors per 100 digits against 0.3 on plain random ones, and 75 of BF16's 80 errors are deletions inside runs of 3 or more equal digits ("eight eight eight eight eight", written one short). That stays unchanged down to about 4.3 bits per weight. Below it, failures turn into insertions, with the model looping on a word. Substitutions stay rare except in builds that collapse.

The original E4 prompt ("write each digit as an English word") failed as a test: the model read it as "say this number" and answered "twenty trillion two hundred…", even at BF16. Those records are kept; E4b adds one example ("507 becomes: five zero seven") and replaces it in the results.

The recipe matters more than the calibration

Each quant type was built several ways: stock with no imatrix, with our digit-free imatrix, with unsloth's imatrix, and unsloth's published file.

• Scheme: at the same size class, legacy Q4_0 has 3.3× the π noise of Q4_K_M.
• Imatrix: ours, calibrated on text with no digit strings, halves generic noise at Q4_K_M but does not protect π; noise on π even rises. An importance matrix protects what its calibration text exercises. At 2–3 bits it matters for everything: it separates a Q2_K that cannot copy from one that can.
• Recipe: stock Q4_K_M with unsloth's imatrix gets little of unsloth's advantage (KL on π 0.041 vs 0.022). Most of the gain is their per-tensor type choices. Q3_K_M shows the same: 0.29 with their imatrix, 0.17 with their file.

Low bits weaken memory's pull

E5 asks for a copy of π or e with one digit changed. Only π position 10 separates the variants: BF16 and almost every build at 4.6 bits per weight or more "correct" it to the memorized digit, while almost every build at 4.4 or below copies the prompt faithfully. This is the opposite of H6, and consistent with H1: quantization erodes memorized content, so it competes less with the prompt.

What it takes to run each build

One Arc Pro B60 (24 GB), llama.cpp SYCL, 8,192-token f16 KV cache. Decode speed is the median the server reported on the test requests (within 0–5% of llama-bench tg128). Prompt speed is llama-bench at 4,096 tokens, since SYCL's per-request overhead makes short-prompt timings misleading.

• VRAM is the file size plus about 770 MiB (steady median during the tests) for every build (KV cache, recurrent state, buffers). A 4 GB card fits everything up to Q5_K_M with an 8k context; BF16 needs about 9 GB.
• Decode stops getting faster below about 5 bits. 38 → 65 → 89 tokens/s from BF16 to Q8_0 to Q4_K_M, then flat to Q2_K. Smaller quants of a 4B model buy memory here, not speed.
• IQ types are slow on SYCL. Their codebook lookups cost more than the bytes they save; UD-XL mixes cost 12–28% decode against plain K-quants of similar size.

Problems found along the way

Determinism checks passed on BF16 and Q8_0: identical bytes on repeated requests, after a server restart, and across both GPUs. Our Q8_0 and unsloth's Q8_0 agree to KL 0.00013, so the pipeline adds no error of its own.

Next

• Phase 3: the 27B, with a Q8_0 reference, to see whether H5's headroom keeps growing.
• Open questions: drop E6? Calibrate the imatrix on "generic text plus some numbers" to make the H4 comparison fairer?