Fine-tuning, with receipts

Fine-tune a small local model on your docs — and prove it actually learned

Every "fine-tune on your own documents" tutorial ends the same way: train a LoRA, run one prompt, eyeball the output, declare victory. This page is the other ending — a portable knowledge pack format where the eval harness ships inside the artifact, plus the real, measured numbers from running it end-to-end on one Mac. Including the run that failed.

1. What a knowledge pack is

A knowledge pack is a versioned, portable directory artifact that compiles a body of expertise into four things:

1. curated sources (license and attribution recorded per source unit),
2. schema-validated generated training JSONL, every record citing the source units it came from,
3. a contamination-guarded held-out eval set, and
4. a JSON manifest — hashes, provenance, generation recipe, base-model compatibility, license.

You "install" a pack by LoRA-fine-tuning a small local model on Apple Silicon (MLX), and you prove it worked by running the same eval harness before and after. The eval harness is not an optional extra — it is the pack's honesty mechanism, and it is the thing every existing tutorial in this space leaves out.

packsmith is the CLI that builds, verifies, evaluates and diffs packs. It is written in Crystal, compiles on the stock compiler, and its deterministic path runs with no model on disk and no API key.

2. The silent failure that motivates everything

Here is a measurement from our reference machine (Apple M1 Max, 32 GB): we trained mlx-community/SmolLM-135M-Instruct-4bit with mlx_lm twice on the identical 471-pair dataset for the identical 2000 iterations — only the learning rate differed. Both per-item eval reports are checked in under live-run/reports/:

• Default learning rate (1e-5): validation loss walked down from 4.94 to 1.96 — a beautiful curve — and recall of trained facts barely moved: 2/20 vs the base model’s 1/20 at temperature 0.
• Same data, same 2000 iterations, lr 1e-4: validation loss reached 0.53 and recall of trained facts went to 11/20 — 62.8 seconds of wall time, 0.42 GB peak memory.

Loss going down is not knowledge going in. If your pipeline can't measure the difference, you don't have a pipeline — you have a ritual. Everything else on this page is machinery for measuring that difference, cheaply enough that you never skip it.

3. The pack format: split before generate, fail closed

Treat "teach a model X" as a build artifact with provenance, and make measurement part of the format itself:

my-pack/
  pack.manifest.json          format + provenance + split recipe + sha256 file table
  sources/curated/*.md        the source material (license + attribution recorded)
  sources/source_index.jsonl  every source unit: raw & normalized sha256, section, license
  generated/train.jsonl       schema-validated training records, each citing its source units
  generated/formats/…         mlx / alpaca / sharegpt rendered views
  eval/questions.jsonl        held-out questions (answer-free view)
  eval/answer_key.jsonl       the answers, quotes and scoring rules
  reports/*.eval.json         before/after eval reports, per-item, published as-is

Three rules make it honest:

1. Split before generate. Source units are assigned to train/valid/eval by a salted hash of the unit id, before any example is generated. Eval questions can only be generated from source sections the training data never saw.
2. Fail closed. The contamination guard — held-out unit-ID exclusion, duplicate-question detection, verbatim-overlap check, n-gram Jaccard threshold — runs at build time and again on verify. Any violation is exit code 5, not a warning.
3. Same harness, both sides. packsmith eval runs the identical question set, prompts and scoring against the base model and the adapter, and packsmith diff --fail-below-delta turns "did it actually learn?" into a CI assertion.

The CLI's exit codes are typed so CI can tell failure modes apart: 0 ok · 1 validation or eval-threshold failure · 2 usage · 3 missing runtime/model · 4 generation failed schema validation · 5 contamination guard · 6 hash/provenance mismatch.

Generation has two honest modes. The deterministic mode (default) is template synthesis from the sources — flashcards, section recall, cloze questions with sibling-term distractors, exact-command probes. Its quality ceiling is stated plainly: it tests memorization and lookup-like recall only, no conceptual synthesis, no troubleshooting judgment. The structured-model mode sends each source unit to a generator model (fully local via MLX, or an API model) with a JSON schema, and parses every reply into typed Crystal classes — wrong enum values, missing fields, or an answer_quote that doesn't appear in the source unit get the record rejected and logged. To be precise about the mechanism: that is schema-prompted generation plus typed validation, not grammar-constrained decoding. Either way, the generated JSONL is the artifact of record; the recorded recipe is reproducibility evidence, not a promise of byte-identical regeneration against remote models.

4. Walkthrough: your docs → pack → adapter, on a Mac

You need stock Crystal (tested on 1.20.0) and, for the training step, a Python venv with mlx-lm. Nothing leaves your machine.

git clone https://github.com/AgentC-Consulting/knowledge-packs
cd knowledge-packs
shards install
crystal build src/packsmith.cr -o bin/packsmith

# the demo pack teaches the Amber web framework (docs condensed with attribution)
./bin/packsmith build packs/amber-framework     # chunk → split → generate → guard → emit
./bin/packsmith verify packs/amber-framework    # hashes, schemas, contamination — fail closed

For your own material:

./bin/packsmith init my-pack --id you.my-docs --name "My Docs"
# drop curated markdown into my-pack/sources/curated/
# (every heading-scoped section becomes one provenance-tracked source unit)
./bin/packsmith build my-pack
./bin/packsmith verify my-pack

Training wraps the measured MLX recipe (scripts/train_mlx.sh):

./bin/packsmith train my-pack --base mlx-community/SmolLM-135M-Instruct-4bit

Then the part everyone skips — the reason this format exists:

./bin/packsmith eval my-pack --model BASE --out reports/base.eval.json
./bin/packsmith eval my-pack --model BASE --adapter ADAPTER --out reports/adapter.eval.json
./bin/packsmith diff reports/base.eval.json reports/adapter.eval.json \
  --out reports/diff.json --fail-below-delta 0.10

5. A measured run — with the control set that keeps it honest

On 2026-07-07 we ran the whole loop on one machine: the Amber framework docs (81 markdown files) → deterministic pack build → baseline eval → LoRA train → re-eval. Everything below was measured in that run; the per-item reports, eval sets, training data and harness scripts are checked in verbatim under live-run/ so you can audit every item.

The build produced 336 heading-scoped source units, split before generation by salted hash into 262 train / 74 eval units, yielding 471 training pairs (each fact trained as two cloze phrasings plus one multiple-choice phrasing). Two 50-item eval sets, each 30 multiple-choice + 20 exact-recall, all scored at temperature 0 with the identical harness on both sides:

• memorization — questions about facts that ARE in the training data, asked through question templates that never appear in training. The underlying fact sentence is shared with training by design: this is a trained-fact probe measuring fact injection, not a source-held-out eval.
• heldout_sources — facts from the 74 eval-split units, which contributed zero training examples (exact-duplicate sentences are deduped into train; semantic restatements elsewhere in the docs cannot be fully excluded). This is the overclaiming control: a memorization-only pack should not improve here.

This run predates packsmith and used a deliberately simple scorer, stated exactly: MC = first standalone A–D letter in the completion, no retry (the packsmith harness specifies JSON parsed to a typed enum with one bounded retry); recall = normalized substring match of the gold term (packsmith specifies exact match or answer-key regex). Applied identically to base and adapter — all deviations are enumerated in RESULTS.md.

Training: 2000 iterations, batch 2, 8 layers, lr 1e-4 → 62.77 s wall, ~33.5 it/s, 0.42 GB peak, val loss 4.939 → 0.530. Results, counts before percentages:

The honest read: fact injection is real and measured — and the control set showed no measurable gain on source units that contributed zero training examples, which is consistent with the adapter learning the facts it was trained on rather than "the Amber framework." (A single 50-item synthetic control bounds that claim; it cannot prove the absence of subtler transfer.) That distinction is the entire point of shipping two eval sets.

The ablations, including the one that broke

• The learning-rate contrast (the silent failure of section 2, measured on this dataset at matched budget): lr 1e-5 for the same 2000 iterations — val loss 4.94 → 1.96, recall 2/20. At 300 iterations (~1 epoch) neither lr shows much (1/20 and 2/20): on this dataset, injection needed both the higher lr and more epochs, and no loss curve predicted any of it. Reports checked in.
• 4000 iterations (123.2 s): memorization 18/50, control 5/50, val loss ticked up 0.530 → 0.546. Diminishing returns; 2000 kept as primary.
• Cloze-only training data (the first attempt, run against an earlier generator revision and its own eval set — hence the different base MC of 8/30; numbers transcribed from the run log): recall improved 1/20 → 7/20, but multiple-choice collapsed 8/30 → 1/30 with 25/30 invalid responses — the adapter answered every MC question with a term instead of a letter. Fine-tuning had damaged format compliance while injecting facts, and no loss curve showed it. Fixed by adding MC-format pairs to the training data. If you take one thing from this page: this failure is invisible to every "eyeball the inference" tutorial.

Caveats — published alongside the numbers, always

1. 135M-parameter 4-bit model: floor-level capability; MC even after training (6/30) is at or below the 25% guessing chance — only the recall probes show clear signal. The headline is the recall delta, not MC.
2. 50-item eval sets: no confidence intervals at this n; counts are reported, percentages are decoration. Our own spec requires 100–200 MC items plus 30–50 hand-reviewed questions for any public headline number — this run's questions are synthetic and not hand-reviewed, so treat these as lab notes and run the harness yourself.
3. Deterministic template generation: tests memorization and lookup-like recall only — no conceptual synthesis, no troubleshooting judgment.
4. The memorization set shares facts (and the underlying fact sentences) with training by design — that is what a fact-injection probe measures; only the question templates are held out. It must never be read as a source-held-out eval. The heldout_sources set is the guard against overclaiming.
5. Base-vs-adapter differences on the control set (recall 2/20 → 0/20) are single-item-scale noise.
6. The Amber framework is MIT-licensed; its docs repository has no explicit LICENSE file, so sources are condensed with attribution recorded in the pack manifest.
7. The run's MC parser accepts the first standalone A–D letter anywhere in the completion, and recall is normalized-substring scored — both more permissive than the packsmith harness spec. Generous to both models equally, stated for reproducibility.

Reproduction commands, artifact sha256 hashes (training data, both eval sets, the adapter), and the full per-item reports are in live-run/RESULTS.md.

6. Eval methodology and what we will never claim

Two probe types produce headline numbers:

• Multiple choice — the model answers as JSON; the reply is parsed into a typed enum (A|B|C|D). Invalid JSON or an out-of-set answer gets one bounded retry that names the failure and repeats the schema; still-invalid scores wrong, and the invalid rate is reported separately, per item. (Watch that invalid rate: it is where the cloze-only format collapse showed up.)
• Exact recall at temperature 0 — commands, class names, config keys, scored by normalized exact match or answer-key regex.

Reports carry counts before percentages, per-item records, and the eval-set hash — diff refuses to compare reports from different eval sets. The only claim shape the format supports:

"On this held-out eval set generated from source sections excluded from training data, the adapter improved measured answer accuracy from X to Y under this harness."

And the fine print, published verbatim because it is true:

This does not turn a small model into a reliable framework expert. It tests whether a local adapter measurably improves source-specific recall and simple usage knowledge under a disclosed eval harness.

Things this project will never claim: general framework mastery, production coding reliability, benchmark superiority, cross-model transfer, or grammar-constrained decoding. The two hardest failure modes are designed against, not wished away: self-flattering synthetic evals (countered by split-before-generate, separate train/eval prompts and schemas, deterministic exact-fact probes, per-item published reports, the identical harness on both sides, and a hand-review requirement before any headline) and silent training failure (countered by making before/after eval the happy path and --fail-below-delta a gate). If an adapter fails to improve its pack's eval, that failure gets published too.

7. Crystal + Llamero: the toolchain under it

packsmith is written in Crystal — a compiled, typed language — on top of Llamero, the Crystal LLM library. That buys three things the Python incumbents don't have:

• Typed validation, not best-effort filtering. Generated records and eval answers parse into real Crystal classes and enums at runtime — the types are declared once, checked by the compiler, and enforced at parse time; an out-of-set answer raises with the raw text preserved instead of slipping through as a string.
• One structured-output interface across native MLX sessions and API clients (OpenAI-compatible, Anthropic, Groq, OpenRouter), so the same generation code runs fully local or against an API.
• A deterministic MockBridge, so the entire spec suite and CI run green on stock Crystal with no model on disk and no API key — the eval plumbing is provable everywhere, including GitHub Actions.

If you searched for a Crystal-language LLM toolchain and found nothing: this is it, and Llamero underneath it is the library the ecosystem was missing.

8. Where this is going

Today: build packs from your own docs and prove them locally. The direction we're building toward — stated as ambition, not as something that exists — is a marketplace of installable, signed knowledge packs: buy the "Amber framework" pack or publish your own "internal-API" pack, each shipping its eval set so every claim is checkable on your machine before you trust it. The manifest's signatures[] field and strict format versioning are already shaped for that future. Image-generation LoRA marketplaces prove people pay for portable model deltas; we have not found a text-model knowledge equivalent with measurement built in. We think measurement is the product.

9. FAQ

What is an LLM knowledge pack?

A versioned, portable directory artifact that compiles a body of expertise into curated sources, schema-validated generated training JSONL, a contamination-guarded held-out eval set, and a JSON manifest (hashes, provenance, recipe, base-model compatibility, license). You install it by LoRA-fine-tuning a small local model; you prove it worked by running the same eval harness before and after. Details above.

Did my LoRA actually learn anything?

You cannot tell from the loss curve — we measured a matched pair of runs (same data, same 2000 iterations) where the default lr walked validation loss from 4.94 to 1.96 while recall of trained facts barely moved (2/20 vs base 1/20), and another run where fine-tuning injected facts while quietly destroying the model's ability to answer multiple-choice questions in the right format. The only reliable check is a held-out before/after eval with the identical harness on both sides, plus an invalid-response rate so format damage shows up. The silent-failure story and the format-collapse ablation are both on this page with numbers.

How do I evaluate an LLM before and after fine-tuning on my docs?

Split your sources into train/eval units before generating any data; generate held-out questions only from eval units; run the identical questions, prompts and scoring against base and adapter; report per-item results with counts before percentages. And add a control set drawn from source units excluded from training-data generation — if accuracy rises there, your eval is measuring pattern-matching, not knowledge. Methodology, worked example with real numbers.

Can I fine-tune a small model on internal docs on a Mac, without the cloud?

Yes — measured on one M1 Max (32 GB): the 2000-iteration LoRA run behind the numbers above took 62.77 s at 0.42 GB peak (~33 it/s; a 300-iteration run takes ~11 s). Your documents never leave the machine, and the deterministic generation mode needs no API key. Walkthrough.

How do I teach a Llama (or any small model) my documentation?

Chunk docs into heading-scoped units; split before generating; train each fact in several phrasings including a multiple-choice phrasing (cloze-only training measurably broke format compliance in our run); use a learning rate and iteration budget that actually inject facts (on our setup, lr 1e-4 at 2000 iterations worked where the 1e-5 default silently didn't); verify with before/after eval plus a control set. Expect measurable recall of trained facts — not framework expertise. That expectation-setting is the fine print, published verbatim.

Is this grammar-constrained training data generation?

No — and we're precise about it. The structured-model mode is schema-prompted generation plus Crystal-side typed validation: the schema travels in the prompt, and every reply is parsed into typed classes, with wrong enums, missing fields, or unanswerable answer_quotes rejected and logged. That's a validation guarantee after generation, not constrained decoding — different mechanism, same practical outcome for dataset hygiene: nothing malformed reaches the training set, and every rejection is on the record.

Is there a Crystal language LLM toolchain?

This is one — packsmith on top of Llamero, which provides native MLX sessions and API clients behind a single structured-output interface, plus the deterministic MockBridge that keeps CI green with no models. More above.

Can you sell a fine-tuned knowledge adapter?

We haven't found an existing marketplace for them — image-LoRA marketplaces prove the monetization pattern, but we know of no text-model equivalent with measurement built in. The pack format is deliberately marketplace-shaped (signatures, strict versioning, eval sets that travel with the pack), and building toward that is our stated, explicitly aspirational direction.