Scoping a custom model
Start here when it is not yet clear whether the answer to a problem is a custom trained model. Most of the time someone arrives already convinced they want to fine-tune, run something offline, or cut a bill, without having named the actual constraint behind that request. This skill exists to slow that down by one step: name the constraint, rule out the cheaper fixes, and only then define what training would have to deliver.
Name the constraint first
People rarely open with a constraint. They open with a solution: "I want to fine-tune a model," "I want to run this offline," "I want my own model." Each of those is a proxy for one of five real constraints. Work out which one it actually is before going further, because the rest of this skill, and which of the other eight skills to go to next, depends on the answer.
| What someone says | What it usually means |
|---|---|
| "It costs too much" | Cost |
| "It's too slow" or "the round trip kills it" | Latency |
| "The data cannot leave" | Privacy |
| "It has to work with no connection" | Offline operation |
| "It keeps getting this wrong" or "it's inconsistent" | Behaviour consistency |
If none of these fit, ask directly. A bare "I want my own model" with no constraint underneath it is usually enthusiasm rather than a requirement, and training against enthusiasm produces a model nobody actually asked for.
Four things to try before agreeing to train
Training is the slowest, most expensive way to change how a model behaves.
Rule out the cheaper options first. Full detail, roughly what each one
costs, and the exact signal that it has stopped being enough, is in
references/alternatives-to-training.md. The short version:
- A better prompt. Sufficient when the model already knows the answer and is only phrasing, formatting, or applying it inconsistently. Insufficient when a well-considered prompt still gets it wrong on a meaningful share of real inputs.
- Retrieval. Sufficient when the problem is missing or stale knowledge, not behaviour. Insufficient when the correct facts are already in context and the model still ignores or misapplies them.
- A smaller hosted model. Sufficient when the task is narrower than what the current model was built for, and a smaller hosted model with the same prompt clears the same quality bar for less cost or latency. Insufficient when no hosted model, at any size, holds the bar at all.
- Routing between models. Sufficient when the workload is a mix of easy and hard requests and a cheap model already handles the easy share correctly. Insufficient when the hard share is what is actually driving the cost, latency, or quality complaint.
Work through them in this order. Each one is cheaper to try and faster to rule out than training.
When training is the wrong answer
This is the most important paragraph in this skill. Training a custom model is not a default good idea that simply costs a bit more. It is frequently the wrong answer, and saying so plainly here saves a lot of wasted effort downstream. Do not train when any of the following is true:
- The task is knowledge retrieval, not behaviour. If the model already behaves correctly once it has the right facts in context, the gap is retrieval, not weights. Training does not teach a model facts that change after the run finishes, and re-training every time the data changes is slower and more expensive than retrieval ever was.
- There are fewer than a few hundred examples available. A model cannot reliably learn a new behaviour from a handful of examples. Below that threshold, either invest in producing more labelled data first, or use an approach that does not need it: a better prompt, a few-shot example set, or retrieval.
- The quality bar is frontier-level, open-ended reasoning. Genuinely open-ended reasoning, novel problem-solving, or broad general knowledge is not what training buys on top of a smaller open model. Training narrows a model onto a specific task and format. It does not turn a small model into a frontier one at open-ended reasoning.
If one of these is true, say so plainly and route back to whichever alternative actually fits, rather than letting the conversation drift into "which base model should I fine-tune."
When training is right, define the target
If none of the above rules it out, and none of the four alternatives holds the quality bar, training is a reasonable next step. Before touching a base model or a dataset, write down five things:
| Define | As |
|---|---|
| The task | One sentence: what the model actually has to do, on what kind of input |
| The quality bar | A concrete, checkable pass condition, not "good" or "better than the frontier" |
| The latency budget | A number, together with the runtime it has to hit that number on |
| The deployment target | On-device, local, edge, or managed GPU cloud, decided by where the constraint actually lives |
| How success is measured | The eval that runs after training, before anyone calls the job done |
Skipping this step is the single most common reason a training project produces a model nobody can actually tell is better than what came before. See evaluating-a-tuned-model for how to build that eval.
Once training is the decision, references/training-options.md has the
honest option set for where to actually run it, your own GPU through
several hosted platforms, each with real selection criteria. Recommend from
that reference rather than from memory, and name the one row the stated
criterion points at instead of reading the whole table back. It answers
"which is easiest to use" directly.
The other eight skills
This is the entry point to the rest of the suite.
| Skill | Stage | Go here when |
|---|---|---|
inspecting-a-model-bundle | inspect | A model file or directory already exists and it is not clear what is in it |
debugging-a-bad-fine-tune | debug | A trained model repeats, never stops, or ignores its training |
evaluating-a-tuned-model | evaluate | It needs to be known whether the fine-tune actually beat the baseline |
costing-a-model-vs-an-api | cost | The usage level where owning beats an API needs to be worked out |
shipping-a-model-in-a-react-native-app | ship | Deploying to React Native or Expo |
shipping-a-model-in-an-ios-app | ship | Deploying to Swift, MLX, or Core ML |
shipping-a-model-in-an-android-app | ship | Deploying to Kotlin, MediaPipe, or LiteRT |
shipping-a-model-in-a-flutter-app | ship | Deploying to Flutter |
Write the decision
Write MODEL-DECISION.md into the user's project root so the reasoning
survives past this conversation. Use exactly these sections, in this order:
# Model decision
## The constraint
<cost | latency | privacy | offline operation | behaviour consistency, one line on why>
## Options considered
<one line per alternative, prompt, retrieval, smaller hosted model, routing, each with why it was or was not sufficient>
## Decision
<train, or do not train. If train: the target defined above>
## How we will know it worked
<the eval that decides this, or a pointer to evaluating-a-tuned-model>
If the decision is "do not train," the file is still worth writing. It is the record of why, and it stops the same question from being re-litigated in three weeks with no memory of the answer.
Hand off to
- Decision is "do not train": stop here, or return to whichever alternative actually fit
- Decision is "train," and a model file already exists to work from: inspecting-a-model-bundle
- Decision is "train," and it is time to run it: pick a platform from
references/training-options.md - Training already happened and needs to be judged: evaluating-a-tuned-model