I tried to make one of my data pipelines cheaper by swapping in a model with a lower per-token price. It cost about four times as much, ran roughly ten times slower, and the first two runs died on a bug that turned out to be my own. The reasons surprised me, so I’ll walk through them.
The pipeline pulls structured records out of large text archives, a chunk at a time. The heavy lifting, reading messy text and returning clean, faithful fields, runs on Claude Opus. Opus isn’t the cheap option, so every few weeks I look at the bill and wonder if I’m overpaying. This time I tested it: I wired in Qwen3.7-Max (the qwen/qwen3.7-max slug on OpenRouter), Alibaba’s flagship and a good bit cheaper per token, and ran the same chunks through both.
The first two Qwen runs crashed, and the cause was my own code. A helper that calls OpenRouter assumed every response body was valid JSON and ran json.loads() on it directly. When the API returned a plain-text error instead, the parser threw and took the whole run down. No guard, no retry. That code had been there for months and never fired, because the only path that reached it had always returned clean JSON. Pointing it at a new provider ran that error path for the first time. I added a guard and a retry and reran.
On paper this should have saved money. It did the opposite. Per chunk, Qwen ran about $0.13 to $0.20; Opus ran about $0.03 to $0.06 on the same input. That’s roughly four times more on a typical chunk. Two things drove it, and neither is on the rate card.
The first was verbosity. Qwen produced about nine times more output per chunk, padding every field with explanation I hadn’t asked for. Output tokens are billed at a higher rate than input, and there were many more of them, so the lower rate disappeared under the extra text.
The second was caching. The shared instructions and context I send with every chunk come to roughly a hundred thousand tokens. Opus caches that prefix: it’s billed once when the cache is written, then read back at about a tenth of the price on every chunk after. Qwen got zero cache reads, so it paid full input price on the whole hundred-thousand-token prefix every single chunk, and that alone is most of what each Qwen chunk cost. Qwen can cache, it has its own prompt cache; I just hadn’t rebuilt that part of the pipeline for a new provider. So the verbosity was the model’s doing. The caching gap was mine, and it’s the only one of the two I could have fixed.
The one place it held up was quality. I had Opus grade both sets of output blind, ten chunks each, no labels, no idea which file came from which model, scoring faithfulness and usefulness on a two-point rubric where 2 is perfect. Faithfulness came out 2.0 for Opus and 1.96 for Qwen; usefulness 1.54 for Opus and 1.63 for Qwen; head-to-head wins split five to five. So across those ten, Qwen was a touch more useful and a touch less faithful. I’d weight this the least of anything here, though: the grader was a Claude model judging a set that included Claude’s own output, and with Qwen running so much longer per answer it could probably tell the two apart anyway.
I kept Opus. Even if I’d closed the caching gap, the cheaper model was still more verbose, about ten times slower, and ahead only on the one soft measure that was never going to decide it, and getting it to run at all had cost me a bug-fix first.
The per-token price, it turns out, barely tells you what a model will actually cost. What you’ve cached, and how much the thing writes, swamp it, and you don’t find out until you run your own load and read the bill.