Fine-tuning, with receipts

Teach a small local model your docs.
Then prove it learned.

A knowledge pack is a versioned, portable artifact — curated sources, schema-validated training JSONL, a contamination-guarded held-out eval set, and a signed-shaped manifest. The before/after eval harness ships inside the format, because loss going down is not knowledge going in.

packsmith build packs/amber-framework      # chunk → split → generate → guard → emit
packsmith verify packs/amber-framework     # hashes, schemas, contamination — fail closed
packsmith train packs/amber-framework --base mlx-community/SmolLM-135M-Instruct-4bit
packsmith eval  … --out base.eval.json     # identical harness, before and after
packsmith diff  base.eval.json adapter.eval.json --fail-below-delta 0.10

Measured, not vibed

One M1 Max, one afternoon, per-item reports checked in. These are lab notes, not benchmarks — the eval questions are synthetic and not yet hand-reviewed, and our own spec forbids headlining synthetic-only numbers. With that stated first: in the measured run, recall of trained facts (normalized-substring match at temperature 0, under held-out question templates) went 1/20 → 11/20, while the held-out-source control set — whose units contributed zero training examples — showed no measurable gain (9/50 → 7/50). Training took 62.77 s at 0.42 GB peak, fully local.

The same experiments also caught two failure modes no loss curve shows: the default learning rate that walks validation loss from 4.94 to 1.96 while recall of trained facts barely moves (2/20 vs base 1/20 — same data, same iterations as the run above), and a cloze-only dataset that injected facts while collapsing multiple-choice format compliance to 1/30. Both are written up with full numbers and caveats — the eval questions are synthetic and not yet hand-reviewed, so read everything here as lab notes and run the harness yourself.

Read the guide

Built in Crystal, on Llamero

packsmith compiles on stock Crystal and runs its whole spec suite against Llamero's deterministic MockBridge — no models, no keys, green in CI. Generated records parse into real Crystal classes and enums — typed validation at parse time, not a best-effort post-hoc filter. Why that matters.