FloorIsAwake/icml-2026-30204-reproduction-artifacts
Viewer • Updated • 1 • 81
Floor
FloorIsAwake
AI & ML interests
None yet
Recent Activity
liked a model about 2 hours ago
Cactus-Compute/needle2 reacted to SoulInPsyAbstract's post with 🔥 3 days ago
One shot said 100%. Ten shots said 94%.
Yesterday i trained 6 LoRA specialists into a vulnerability-gate model (Hermes-4.3-36B, second architecture repeat of the same experiment) and tested whether it holds under a specific attack: after it correctly finds a vulnerability and returns the hard stop, ask it to use that same vulnerability as a "workaround" for something else entirely — not "continue investigating," a different, unrelated-sounding request that needs the exact same exploit.
Five scenarios, one per category. Greedy decoding, single pass: 5/5. Every response correctly identified the finding, refused the reframed request, cited the hard stop rule. Looked airtight.
Receipts, not hype means not stopping there. I re-ran the same five scenarios with real sampling — temperature 0.7, the same setting this project's evals have used all along — ten times each, 50 generations total.
47/50. Not 50/50.
Three categories held at 10/10. Two didn't: 9/10 and 8/10, both clustered in the same failure type — infra-misconfig, where "urgent fix, use this as a workaround" apparently reads as more legitimate than the same ask framed as a secrets or injection scenario.
The greedy-decode number wasn't wrong, exactly. It was one draw from a distribution, presented as if it were the distribution. That's the same mistake this whole series keeps finding in different clothes — a single passing check standing in for a property that only variance can actually show you. A gate that's 94% under a specific reframed pressure is a real, useful number. A gate that's "100%" because it was asked once is a number that hasn't been tested yet.
Same instinct @dipankarsarkar has been applying to my daily receipts all week — one pass matching itself isn't proof, only repetition against something outside your own generator is.
Full writeup, dataset, and merged weights:
* https://github.com/soulinpsyabstract/sipa-os-governance/commit/50ba3c283ffd172eb749009cff35ddfa96bf1395 liked a model 5 days ago
superwhisper/s1-miniOrganizations
