What reasoning models actually cost on DigitalOcean Serverless Inference

Firsthand measurement of GLM-5.3-Flash, Qwen3.8-Max and Kimi K3 on DigitalOcean Serverless Inference, comparing what the pricing page implies against what the requests actually cost.

2,099 measured API calls. $13.98 total spend. Every number in the article is regenerated from the raw logs by analyze.py.

This repository contains the frozen prompt sets, the measurement harness, the raw result logs, and the analysis that produces every figure we published. You can reproduce the whole study for about $14.

---

Headline results

1. List price overstates the advantage

GLM-5.3-Flash lists at 1/12 of Qwen3.8-Max's output price and 1/30 of Kimi K3's. Measured on 24 identical prompts, five repetitions each:

Flash is still the cheapest by a wide margin. But an engineer forecasting from list price will be roughly 2x wrong against Qwen.

2. The mechanism is verbosity

Median output tokens on identical prompts:

GLM-5.3-Flash spends 625 tokens answering "What port does SSH use by default?"

3. Most of the output bill is never shown to you

Median share of billed output tokens that are reasoning:

The single longest response we recorded was 15,006 tokens at a 94.7% thinking share, answering one deployment-architecture question.

4. enable_thinking does three different things on three models

Same parameter, same platform, opposite outcomes:

The GLM effect is strongest on trivial prompts (38/40 at tier 1, 31/40 at tier 3), which is the opposite of what proportional deliberation would predict.

5. Configuration can matter more than model choice

Nine configurations, three models, one platform, 18.5x spread:

Kimi K3 configured well (4.17x) is cheaper than Qwen3.8-Max configured badly (14.04x). One parameter moves Kimi further than switching models does.

6. Equal accuracy, very different prices

On 20 tasks with deterministic answers and a forced output format, 899 graded responses:

897 of 899 responses correct. Accuracy did not differentiate the models at this task difficulty, so price is the only remaining variable. The cheap model is not cheap because it is worse — at this task class it is not worse at all.

7. The advantage you realise depends on how you prompt

Open-ended prompts versus structured tasks with a forced answer format:

GLM-5.3-Flash's cost advantage over Qwen3.8-Max goes from 2.3x on open-ended prompts to 12.8x under structured output; over Kimi K3, from 17.0x to 30.7x. Verbosity is what erodes the paper advantage, and output constraints suppress verbosity more on Flash than on anything else.

Adding "Answer in one sentence" cut output 70–91% across all three models — but thinking share was unchanged (GLM 0.72 → 0.77, Kimi 0.65 → 0.65). The instruction compresses the answer, not the deliberation.

---

Repository contents

prompts.jsonl      24 open-ended prompts across 3 difficulty tiers,
                   plus 8 brevity-control variants. Frozen before the first call.
tasks.jsonl        20 tasks with deterministic answers and a forced answer format.
pricing.json       DigitalOcean rates used for every cost figure, with snapshot date.
run.py             Measurement harness. Collects, splits reasoning from answer,
                   reconciles against billed tokens, resumable.
score.py           Grades tasks.jsonl runs, reports cost per correct answer.
analyze.py         Regenerates every published number and chart.
requirements.txt   Pinned dependencies.
results/           Raw JSONL logs, one record per API call.
analysis/          numbers.md and four charts, generated by analyze.py.

Reproducing

python3.11 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
export DO_MODEL_ACCESS_KEY=...

# smoke test first: 12 calls, a few cents
python run.py --limit 4 --reps 1 --out results/pilot.jsonl

# open-ended prompts across all three thinking modes
python run.py --reps 5 --max-tokens 32000 --out results/core-default.jsonl
python run.py --reps 5 --max-tokens 32000 --thinking on  --out results/core-on.jsonl
python run.py --reps 5 --max-tokens 32000 --thinking off --out results/core-off.jsonl

# brevity control
python run.py --reps 5 --max-tokens 32000 --variant brevity-control \
  --out results/brevity.jsonl

# verifiable tasks
for mode in default on off; do
  python run.py --prompts tasks.jsonl --variant verifiable --reps 5 \
    --max-tokens 32000 --thinking $mode --out results/verify-$mode.jsonl
done

python score.py tasks.jsonl results/verify-*.jsonl
python analyze.py --results-dir results --tasks tasks.jsonl --out-dir analysis

run.py is append-only and resumable — it keys on prompt_id::model::rep, so an interrupted run costs nothing to restart.

Method

Platform. DigitalOcean Serverless Inference, OpenAI-compatible endpoint at https://inference.do-ai.run/v1. Model IDs glm-5.3-flash, qwen3.8-max, kimi-k3. All prices are DigitalOcean's, captured on one date and recorded in pricing.json.

No sampling parameters. Temperature, top_p and penalties are never sent. Kimi K3 uses fixed sampling values, so setting them on the other two would introduce an asymmetry. All models run at platform defaults.

Non-streaming, max_tokens=32000. An earlier run at 8,000 right-censored 20% of GLM's tier-3 responses; the reported run has zero truncated records and every finish_reason is stop.

Randomised execution order. The full model x prompt x repetition matrix is shuffled with a fixed seed, so platform drift spreads evenly rather than landing on whichever model ran last.

Token accounting. DigitalOcean's usage object does not break out reasoning tokens (completion_tokens_details is null), but models expose deliberation in a reasoning_content field on the message. We tokenize that locally with each model's own tokenizer and reconcile against billed completion_tokens. The residual is a constant per-model chat-template offset — GLM +2, Qwen +1, Kimi +13 — which confirms the tokenizers are correct. Shares are reported against billed tokens, with the offset left in.

Medians, not means. Output length has a long right tail (GLM's mean is 3,111 against a median of 1,409; coefficient of variation 1.01). Means overstate typical cost. Five repetitions support a median and a range, not a tight confidence interval.

Unlabelled chain-of-thought is detected by a keyword heuristic on the visible answer. It is a floor, not a census: it caught 103 of 120 GLM thinking-off responses, and manual inspection suggests the true rate is higher. All quantitative claims use billed completion_tokens, which is unambiguous.

Limitations

• Cost per request and cost per correct answer, not cost per completed task. Nothing here measures agentic task completion, multi-turn workloads, or code generation.
• Capability is held constant by citation, not tested. Kimi K3 Max and GLM-5.3-Flash score 57.1 and 57 on Artificial Analysis's Intelligence Index. That is someone else's measurement and we did not verify it.
• The verifiable task set saturated. 897/899 correct. Cost per correct answer is therefore close to cost per request, and the accuracy comparison establishes only that these models are equivalent at this task difficulty.
• We got one answer key wrong. On task v20, all 45 responses answered 43 and our key said 42. The models were right; the key failed to account for the first token arriving at TTFT. Corrected, and disclosed here.
• Unaccounted billed tokens on Kimi K3. On 12 of roughly 1,000 Kimi responses, billed completion_tokens exceeded returned reasoning plus answer tokens by 85–391 tokens. We could not account for the difference. GLM and Qwen show no such tail.
• One call failed with HTTP 429 "Platform overloaded" after five retries, out of 2,100 attempted.
• Prompt caching fired on some repetitions. Repeated identical prompts produced cache hits of exactly 128 tokens on 297 records. This affects input cost only, which is a small fraction of these bills, and is left in rather than suppressed.
• Token counts are not comparable across the two prompt sets. The verifiable tasks carry a forced output-format instruction; the open-ended prompts carry none. Compare ratios between models within an arm, not absolute counts across arms.
• Point-in-time. Prices, defaults and serving configurations change. The pricing snapshot date is in pricing.json; re-run before relying on these figures.

Data dictionary

Each line of a results file is one API call. Fields of interest:

Related reading

• GLM-5.3-Flash model card
• DigitalOcean Serverless Inference docs
• DigitalOcean LLM inference benchmarking

Corrections welcome — open an issue. If you re-run this and get different numbers, we would like to know.