Custom ML Model vs LLM API: When to Train Your Own
The default in 2026 is to call an API and move on. It is usually right. But there is a large class of problems where a small model you own is 100× cheaper, 10× faster and more accurate — and teams reach for the API anyway because nobody told them.
- Published
- Updated
The decision table
| Problem | LLM API | Custom model | Usually pick |
|---|---|---|---|
| Open-ended text generation | Excellent | Poor unless huge | API |
| Classification with 10k+ labelled examples | Good, expensive at scale | Excellent, near-free | Custom |
| Extraction from structured documents | Good | Excellent after fine-tune | Custom at volume |
| Recommendation / ranking | Weak | Excellent | Custom |
| Forecasting, anomaly detection | Not applicable | Excellent | Custom |
| Vision on your own imagery | Good, costly per image | Excellent, cheap | Custom at volume |
| Low-volume, changing tasks | Excellent | Overhead not worth it | API |
| Strict data residency | Needs enterprise contract | Runs on your metal | Custom |
The unit economics
Take a support-ticket classifier handling one million tickets a month. Through a frontier LLM API at ~500 tokens per call, that is roughly $1,500–$5,000 a month, forever, with 1–3 s latency. A fine-tuned small model on the same task costs $8k–$20k to build, runs on a $150/month instance, answers in 30 ms and — with 20k labelled tickets — is more accurate, because it has seen your labels and the API has not.
The crossover is usually between 100k and 500k predictions a month. Below it, pay the API and spend your engineering on product. Above it, the custom model pays for itself inside two quarters.
The hybrid most teams should build
The best production systems we ship are rarely one or the other.
- Use the LLM API to label your first 5–10k examples, with human review on a sample. This turns weeks of annotation into days.
- Train a small model on those labels. Route the confident 90% of traffic to it.
- Send the uncertain 10% to the LLM, log the answer, and fold it back into the training set monthly.
- Keep an eval set that never changes so you can see whether each retrain actually helped.
This pattern gives you API-level coverage on hard cases and custom-model economics on the bulk, and it compounds: every month the small model gets a little better and the API bill gets a little smaller.
What you need to train your own
- Data: a few thousand labelled examples for classification or extraction; tens of thousands for anything generative.
- A clear metric that maps to business value — not "accuracy" but "tickets mis-routed per thousand".
- A deployment target you can monitor. A model nobody watches drifts silently.
- Someone accountable for retraining. Models are not software; they decay.
Not sure which side of the crossover you are on? Send us the task and your monthly volume through the form below. We will run the numbers and tell you whether to train, call or hybrid — before you spend anything.
Frequently asked questions
Does fine-tuning an LLM count as a custom model?
It sits between the two. Fine-tuning a hosted LLM improves format and tone cheaply but keeps API pricing and latency. Fine-tuning an open-weights model you host yourself gets you the custom-model economics.
How much data do I really need?
For a classifier with a handful of classes, 2–5k clean examples typically beats a prompted LLM. Quality matters more than quantity — 2k consistent labels beat 20k noisy ones.
What about data privacy?
A model you host never sends data outside your infrastructure. For regulated data that alone can decide the question.
