Key takeaways
The cost of inference at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024, from about $20 to about $0.07 per million tokens.
In a typical document workflow, the model accounts for a small share of variable cost. In our illustrative example it is about 3 percent; reviewing flagged items is the rest.
Review cost scales with the exception rate, so a lower exception rate is worth more than a cheaper model. A one-point fall in flagged items saves more than the entire inference bill.
The measure that matters is cost per completed task, including review and rework, compared with the cost of doing the same task before.
The price collapse
Stanford’s AI Index estimates that the inference cost of a system performing at the level of GPT-3.5 fell more than 280-fold between November 2022 and October 2024.1 Measured per million tokens, the cheapest qualifying model went from about $20 to about $0.07.
Note: GPT-3.5 level is defined by the AI Index as 64.8 percent accuracy on the MMLU benchmark.
Source: Stanford HAI, AI Index Report 20251.
Adoption has followed price. Flexera’s 2025 survey found 72 percent of organizations using generative AI public cloud services extensively or sparingly, up from 47 percent a year earlier.2 For most use cases, the question is no longer whether a model is affordable.
Where the money goes
A business case for a generative AI workflow should count every cost that rises with volume. Inference is one. The others are the time people spend checking output, correcting errors that pass the check, and running the integration and monitoring that keep the system in service.
To show the proportions, consider a document-intake workflow processing 1,000 invoices. Each invoice uses about 6,000 input and 800 output tokens, at a blended price of $1 per million tokens. One in ten documents is flagged for a person to review, which takes four minutes at a loaded cost of $30 an hour.
Note: Illustrative. Assumptions are stated in the text; they are not client data. Integration, hosting and monitoring are excluded because they are largely fixed.
Source: Corvx calculation.
The figures will differ by case, but the shape rarely does. Halving the token price saves $3.40 per thousand documents. Reducing the share of flagged documents from 10 to 9 percent saves $20.
The cost that scales is review
Review cost depends on two numbers: how often the system asks for help, and how long each request takes to resolve. Both are design decisions. A system that flags documents with a reason, shows the source field beside the extracted value and pre-fills the likely correction takes a fraction of the time of one that simply marks a document as uncertain.
Errors that are not flagged cost more. An invoice posted with the wrong amount is found later, by someone else, with less context. That is why the exception rate should never be lowered by raising the confidence threshold alone. The aim is fewer cases that need review, not fewer cases that receive it.
Halving the token price saves $3.40 per thousand documents. Flagging one percent fewer documents saves $20.
Measuring the right unit
Spend on tokens is easy to report and says little about value. The useful unit is the cost per completed task: everything spent on the workflow in a period, divided by the tasks that left it finished and correct.
Measured this way, the choice of model becomes a tuning decision inside a larger budget. A more expensive model that halves the exception rate is usually the cheaper option.
Four questions for a business case
- What does a completed task cost today?Measured on real volume, including the time spent correcting errors downstream.
- What share of items will need a person?Estimated from a labeled sample of production data, not from the pilot’s best run.
- How long does each review take?Timed with the actual review interface, by the people who will use it.
- What happens to errors that are not flagged?Who finds them, how late, and at what cost to correct.
References
- 1.Stanford Institute for Human-Centered AI. AI Index Report 2025. hai.stanford.edu
- 2.Flexera. “New Flexera Report Finds that 84% of Organizations Struggle to Manage Cloud Spend” (2025 State of the Cloud Report). Press release, 19 March 2025. flexera.com
