Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
appvoid 
posted an update 1 day ago
Post
126
A Small Model is All You Need. Meet palmer-006 (90M)

After 3 years of experiments, we are finally releasing our flagship tiny model: **palmer-006**.

If you are building for edge hardware, SBCs (Raspberry Pi, etc.), or low-power devices, this is for you. Inspired by Andrej Karpathy's idea of a self-contained "cognitive core," we wanted to see how much power we could pack into a sub-100M parameter footprint.

📊 **The cool part:**
As of today, palmer-006 takes the **🥈 2nd spot on ArithMark-3**, sitting only behind MobileLLM-R1-140M-base... while being **36% smaller**. It's a 90M parameter model hitting as a highly competitive baseline against 125M+ models.

🧠 **How we "Palmerized" it:**
We believe in starting our experiments with the absolute strongest baseline possible.
1. Light fine-tuning on highly curated data
2. Model merging
3. Another light fine-tuning round
4. Adjusted Mamba for maximum token speed ⚡️

⚠️ *Note: This is a foundational language model. It has not been instruction-tuned yet!*

Also, since this needs instruction tuning next to become a chat assistant—**what dataset would you recommend we use for the instruct tune?**

---
🔗 **Quick Links & Info:**

* **License:** Open for research, education, hobby, and modification! (For commercial use/hosted APIs, shoot an email to nosoyhackercodigo@gmail.com. *PS: Donators can claim a free commercial license!*)

* **Attribution:** Built using AI tech from the Technology Innovation Institute (TII).

Can't wait to see what you build at the edge. Let me know your prompt completions below! 👇

appvoid/palmer-006
appvoid

Can't wait to see what the community 🪐do with this! 🚀🚀🚀
dipankarsarkar

Before the dataset question I would fix the eval question. What makes palmer-006 interesting is 2nd on ArithMark-3 at 90M, and at that size an instruct tune is not a free layer on top. It competes for the same capacity that earned the rank.

So whichever dataset you pick, the number worth publishing is ArithMark-3 before and after. If it survives, the dataset choice is validated by something other than vibes. If it drops 15 points, you learned more than a chat template would have given you.

Have you run the base eval a second time post-merge? Curious how noisy that ranking actually is at 90M.

·
appvoid
•
edited about 2 hours ago

That’s a fair point. At this scale, instruction tuning is definitely a capacity tradeoff rather than a free capability layer, so preserving the base model’s strengths will be one of the main acceptance criteria.

Regarding the post-merge result, we have done additional validation runs and the performance appears directionally consistent, although I would prefer to publish the full repeated-evaluation results once the methodology and comparison conditions are finalized. At 90M, even small evaluation details can meaningfully affect the reported ranking.

The instruct version will be treated as a separate checkpoint rather than a replacement for the base model.
dipankarsarkar

Separate checkpoint is the right call, and it makes the number cheaper to publish, not more expensive.

If base and instruct are two checkpoints rather than one replacement, the base variance does not depend on the instruct methodology at all. Five seeds of the same ArithMark-3 config on palmer-006 alone gives you the noise band at 90M, and that band is the denominator every later comparison needs. It stays valid no matter what the instruct recipe ends up being.

Right now a reader cannot tell whether 2nd place is a two point lead or a two point coin flip. That is the gap the spread closes, and it closes before the instruct tune exists.

When you say the results are directionally consistent, is the variation you see across seeds, or across harness details like prompt format and answer parsing?

·
appvoid

Mostly across harness details rather than a completed five-seed sweep. We have checked the checkpoint under repeated runs and small evaluation-pipeline variations, but I do not want to represent that as a formal variance estimate.

Your distinction is valid though, prompt formatting and answer parsing measure harness sensitivity, while multiple seeds under one frozen configuration might establish the actual noise band. I don't know if it will be published at all. Since is a little bit chaotic keeping base skills while improving them so our iteration method is barely reproducible.
dipankarsarkar

Reproducible training and reproducible eval are different problems, and only the second one blocks the number.

The iteration loop can stay chaotic. Nobody was going to reproduce a 90M finetune search anyway. What has to be frozen is the eval side: prompt format, parsing rule, seed list. Freeze that once and every checkpoint you try afterwards gets measured on the same ruler, including the ones you throw away.

That is worth more to you than to a reader. A chaotic loop plus a moving harness means you cannot tell your own improvement from harness drift, which is probably why the runs read as directionally consistent rather than as a number.

Is the ArithMark-3 harness pinned anywhere you can re-run, or is it re-derived each time you evaluate?

·
appvoid

Yes, I got the script for ArithMark-3 and I can re-run it with no issues.
simonko912

Comparing this to my 0.2b (instruct) model proof of concept trained on 200mb, this 90m model is great!

Screenshot_2026-07-28-21-45-46-694_com.android.chrome-edit
(If you want i can try same benchmarks on your model, also what did you use for benchmarking?)
In this post