VIDRAFT_LAB

SuperGPQA is in. Same 1,000 held-out questions, 4 samples each, parent UD-Q4_K_XL vs POCKET, same settings, 16K generation cap. The only bytes that differ are the 300 Q8_0 tensors.

Accuracy (mean of 4):

• parent 59.10%, POCKET 61.55%
• paired difference +2.45 points [+1.50, +3.35]
• single sample 58.70 vs 62.20, majority of 4 63.40 vs 65.40

The cap matters here, so up front: 806 of the parent's 4,000 samples hit 16K, vs 688 for POCKET, and most capped samples give no answer (796 vs 675). On the 667 questions where no sample in either build hit the cap, the difference is +0.64 [0.00, +1.31].

So most of the gain is POCKET finishing within the budget where the parent runs out. Same budget, more answers delivered.

Length: mean 6,716 to 6,051 tokens (-9.9%), geometric-mean ratio 0.868 [0.848, 0.888]; on the uncapped questions the token-weighted ratio is 0.842.

Symmetric deciles (ratio / accuracy parent vs POCKET):
D1 0.91 (82/82), D2 0.91 (76/76), D3 0.83 (80/80), D4 0.80 (74/75), D5 0.79 (81/83), D6 0.81 (74/74), D7 0.86 (62/63), D8 0.86 (44/54), D9 0.95 (19/27), D10 1.00 (1/2)

Two differences from MMLU-Pro. First, the bottom deciles also shrink (about 9%), probably because even the "short" SuperGPQA traces here are 500 to 1,000 tokens, past the ~700 step we saw. Second, D9 and D10 sit at the cap in both builds, so their ratios are compressed toward 1.0, and D8 and D9 are where POCKET's earlier finish turns into the accuracy gap.

The greedy first-commit run is next; we will post it here.

That is a clean way to split it, and the re-verification hypothesis fits what we see: short answers barely change, long ones drop by a step.

We will run it as you describe: greedy decoding on the same prompts, parent vs POCKET, on a sample from deciles 6 to 10 plus a small control from deciles 1 to 5. For each question we will report where the first divergent token sits relative to the first answer commit, whether both builds commit to the same first answer, and how many tokens follow that commit in each.

Results will go up here together with SuperGPQA.

Good question. From the symmetric cut, deciles 6 to 10 (geometric-mean ratio, roughly 700 to 78K tokens):

D6 0.872, D7 0.867, D8 0.857, D9 0.741, D10 0.876

So it looks more like a step than a slope. The ratio drops from about 1.00 to about 0.87 around 700 tokens, then stays in the 0.74 to 0.88 range without a steady decline; the longest decile (D10) comes back up to 0.876. D9 is the one dip, and with ~200 questions per decile we would not read a trend into a single bin.

We will check whether the same step shows up on SuperGPQA, where many more traces run past 700 tokens, and include the per-decile table with the cap counts.

Thanks for reconciling the cells. Here are the deciles, with one caveat about how to cut them.

Cut by parent length, as you asked, the top decile does fall: POCKET/parent ratio 0.756 [0.687, 0.829] (geometric mean 0.611), vs 1.008 for the bottom nine combined.

But that cut is biased. Sorting on one build's length puts the questions where that build happened to run long into the top decile, so the ratio there is pulled down by regression to the mean. Cut the same data by POCKET length and the top decile flips to 1.027 (geometric 1.177).

So we sorted on the geometric mean of the two lengths, which treats both builds the same:

• Deciles 1 to 5 (up to ~700 tokens): ratio 0.97 to 1.03, essentially unchanged
• Deciles 6 to 10 (~700 tokens and up): 0.74 to 0.89, i.e. 11 to 26% shorter
• Top decile vs bottom nine: +0.088 [-0.009, +0.187] (mean ratio), -0.041 [-0.12, +0.043] (geometric). No separate drop in the top decile.

So it is neither a top-decile effect nor a flat ~15% shrink. Short answers stay the same length; answers past roughly 700 tokens get 11 to 26% shorter. The typical per-question reduction (geometric mean ratio) is 8.7% [6.2, 11.2]; the 14.5% headline is the token-weighted total, carried by the long questions. Accuracy tracks the parent within ±2 points in every decile.

For SuperGPQA we will report both the parent-length and symmetric cuts, plus how many samples hit the 16K cap in each build, since capping can shrink the ratio. Results when both builds finish.

Thanks, these are the right questions. All numbers are from the same 2,000 MMLU-Pro questions, paired, parent UD-Q4_K_XL vs POCKET, same settings (T 1.0, top-p 0.95, top-k 20, seed 7, 131K cap, llama.cpp b11048).

Q1. POCKET used fewer tokens on 1,088 questions, more on 903, same on 9. Mean difference -628 tokens [-871, -383]. Median difference -16 [-22, -7]. So most short answers barely change; the saving comes from the long reasoning traces.

Q2. Split by correctness (parent / POCKET):

• both right, 1,708 questions: 2,926 to 2,530 tokens, -395 [-589, -195]
• parent right, POCKET wrong, 51: -773 [-3,396, +1,890], not significant
• parent wrong, POCKET right, 45: -2,808 [-6,992, +1,087], not significant
• both wrong, 196: -2,117 [-3,421, -866]
  The shortening is mostly in questions both builds got right. There is no sign that POCKET gets shorter by giving up on the ones it misses.

Q3. Zero runs hit the 131K cap in either build, so capped runs do not move the mean. One parent run produced no extractable answer, none for POCKET.

So on this set: same accuracy, and the same correct answers reached with fewer tokens, with the reduction concentrated in long traces. The SuperGPQA R0 vs R3 comparison on the 1,000 held-out questions is running now; we will post it here when both builds finish.

Thanks for verifying the graft byte by byte. That is exactly the check we hoped someone would run.

You were right about shard 1's metadata. It now reads "POCKET-Darwin-180B" on both Hugging Face and ModelScope (only that 10 MB metadata file changed, the three weight shards are untouched).

On your question: the parent UD-Q4_K_XL build scored 87.95% on the same 2,000 questions, so on MMLU-Pro the parent, R3 BF16 (87.65%) and POCKET (87.65%) are all within sampling noise. The ~100 flips you estimated are that noise floor: at temperature 1.0, any two of our builds flip 85 to 98 of the 2,000 questions, including R1 vs R3 in BF16.

MMLU-Pro is close to saturated for this family, so the RSI delta shows on held-out SuperGPQA (+1.03, lower bound +0.05). Your framing of POCKET vs the parent build as a clean R0 to R3 test, with the quantized experts fixed byte for byte, is a good one. We are running exactly that now on the same 1,000 held-out SuperGPQA questions (4 samples each) and will post the numbers here.

One thing the MMLU-Pro run already shows: at the same accuracy, POCKET answers in 3,694 tokens on average vs 4,322 for the parent build, about 15% shorter.

Thanks! "Pocket" here means it fits on one personal machine instead of an 8-GPU server. For smaller laptops we already have POCKET-35B-GGUF (669K downloads) and POCKET-26B-GGUF (296K downloads), and a smaller Darwin RSI build in POCKET form is on our list. We will post here when it is up.