Self-hosting vs. managed inference: where the cost curve actually crosses
"Self-hosting is cheaper" is only true above a certain utilisation. Below it, you are paying for an idle GPU and an engineer's attention — and that second line item is the one nobody puts in the spreadsheet.
What happened
Teams routinely compare a managed API's per-token price against the hourly cost of a GPU instance, conclude self-hosting wins, then discover their average utilisation is under 30% and someone has to patch drivers.
Why it matters for deployment teams
The honest comparison has three terms: compute at your real utilisation, the engineering time to run and upgrade the stack, and the cost of the latency or capacity ceiling you inherit. Managed providers are effectively selling you the second and third items at a markup.
What to do about it
Instrument utilisation before deciding. If your p50 utilisation is below roughly 40%, start managed and revisit quarterly — reserved capacity and better serving stacks keep pushing the crossing point outward.
What to do about it
- Measure real GPU utilisation before committing
- Price engineering time, not just compute hours
- Re-run the comparison every quarter