Most teams pick a model by scanning dollars per million tokens. The AWS Machine Learning Blog argues that number is the wrong unit of comparison, because production workloads don’t buy tokens — they buy a resolved ticket, a finished research brief, a summary a reviewer signs off on. In a September 11, 2026 post, the team walks through an open-source harness that measures the multipliers the pricing page hides: how often the model is right, how many tokens it burns getting there, and how many turns an agent needs.
What the harness actually measures
The benchmarking harness runs one identical code path — the OpenAI Responses API — against five models: gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol on Amazon Bedrock, plus gpt-5.4-mini and gpt-5.4-nano on the OpenAI API as cost-optimized baselines. The AWS authors are explicit that this compares practical deployment configurations, not intrinsic model capability: the Bedrock models ran with reasoning disabled, while the API baselines ran at their defaults.
Three measurements matter. Single-call accuracy and cost on AIME, GPQA Diamond, and MMLU-Pro. Multi-turn agent trajectories on live web-research tasks. And rubric-graded professional deliverables from a 48-task GDPval slice. Grading mixes deterministic checks with an LLM judge, and every number is generated from timestamped result files.
Cost per correct answer, not cost per token
For each benchmark, the harness divides total spend — right and wrong attempts alike — by the number of correct answers. On AIME, sol solves 75 percent of problems to mini’s 37 percent, and leads on GPQA Diamond (68 versus 43 percent) and MMLU-Pro (82 versus 59 percent).
The more interesting finding is what token efficiency did before any price moved. At its original list price, luna was already 25 percent cheaper per correct AIME answer than mini in this configuration, because with reasoning disabled it used fewer billed tokens. After the July 30, 2026 price reduction for GPT-5.6 Luna and Terra on Amazon Bedrock — luna down 80 percent, terra down 20 percent — the recorded cost per correct AIME answer is $0.0021 for luna versus $0.0139 for mini. Luna also beat nano on that metric despite nano’s slightly lower nominal token price.
One caveat the authors flag: sample sizes run 48–198 items, so treat small gaps as directional.
Turn count is a pricing variable
Agentic workloads break single-call math. The harness uses client-managed history with store: false, so every turn re-sends the system prompt, prior tool results, and accumulated context. Per-turn context grows roughly linearly, which means cumulative billed input grows close to quadratically with turn count.
On a 50-question DeepSearchQA sample with real web_search and fetch_page tools, mini took the most turns per question (7.6, mostly re-search loops) and reached 2.3× terra’s input-token volume — 114k versus 50k tokens per question. Terra’s higher token price was offset by fewer turns and higher quality: $0.31 per passing answer versus mini’s $0.40, with mean F1 of 0.50 versus 0.39. Luna came in at $0.05 per passing answer against mini’s $0.40. Nano’s lower token price didn’t save it: at an 18 percent pass rate, its observed cost per passing answer was $0.07.
If your agents chain tool calls — research, multi-hop lookups, iterative retrieval — trajectory cost belongs in the same spreadsheet as call cost. This is the same discipline behind prefix-aware routing on SageMaker: the shape of your traffic, not the sticker price, decides the bill.
Where the premium is still real
GDPval is the closest thing here to shipped work: real occupational deliverables graded against human-authored rubrics, passing at 70 percent of weighted points. All three gpt-5.6 configurations scored higher than mini and nano with reasoning disabled, with the largest gaps in law, nursing, and financial advice. Luna passed 27 of 48 deliverables to mini’s 20, at $0.010 per passing deliverable versus mini’s $0.030 and nano’s $0.012.
Terra and sol still cost more per pass while scoring higher on rubrics. Whether that premium pays off depends on what review and rework cost in your workflow — a number only you have.
What to do with this
The AWS authors’ own recommendation is to run the harness on your own tasks before deciding. That’s the honest takeaway: their samples are small, latency figures are a point-in-time regional snapshot, and the GDPval outputs were capped at 8,192 tokens, truncating several deliverables. A higher cap might improve quality and cost together, so test both.
If you’re on mini or nano today, the useful first step isn’t a migration — it’s instrumenting cost per successful outcome on your real workload, turns included. The pricing page won’t tell you which model wins.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.