It's Time to Deploy: Own Your Intelligence

July 29, 2026 • 7 min read • Arkadij Kummer

#LLM #Benchmarks #AI #Kimi K3 #Open Weight Models #vLLM #Self-Hosting #Inference #Reasoning Models #Structured Output #GPU #Evaluation #SEC Filings

Kimi K3 shipped on 27 July: 2.8 trillion parameters, 16 of 896 experts active per token, and an official 4-bit release that was quantization-aware trained. 2 days later we had it running on Modal, 8 rented B300 GPUs at $56.79 per hour with per-second billing, and we spent a day measuring what self-hosting a frontier-class model actually buys.

What it buys starts before any benchmark. The weights are yours, pinned: the model can't be deprecated, silently updated, or rate-limited underneath you, and the same files run on any provider today or your own racks later. The data path shortens too: on a rented box your documents flow to 1 infrastructure provider you chose, under your configuration, rather than to a model vendor plus whoever routes in between. Full custody still means your own hardware, whose economics we didn't measure here. And the serving configuration is yours, which cuts both ways, as the 0.19-recall trap below shows. Owning intelligence stopped being a compromise somewhere around GLM-5.2, which beat GPT-5.5 on green flags in our June eval. K3 raises the ceiling of what you can hold.

It might even be cheaper. At saturation, the box produces output tokens at $3.01 per million against the API's $15.00, processes fresh input at $0.73 per million against $3.00, and streams a single response 3 to 5 times faster than the API serves the same weights. On our SEC filing benchmark the self-hosted model scores within noise of the vendor's own API, and above GPT-5.5 and Sonnet on 1 of the 2 tasks.

The cheaper part carries a condition: saturation. The same box at our benchmark's own utilization cost about 14x more than the API per token. Renting frontier hardware pays exactly when you can keep it busy. This post covers what that takes, what the box delivers on quality, speed, and cost, and the serving trap.

What the box can actually do

Serving the model ourselves produced numbers no API exposes.

Single-stream decode runs at 77 to 101 tokens per second depending on config. The API streams the same model at about 32 tokens per second on public trackers, and about 20 in our own runs, with a 4-second median time to first token and a worst call of 616 seconds. The identical red-flag run took 34 to 38 minutes self-hosted and 57 to 72 minutes through the API.

We then load-tested the deployment to find its ceilings, raising concurrency until throughput went flat across consecutive levels, with achieved parallelism verified at every step.

USD per million tokens at saturation · log scaleself-hosted 8x B300 (Modal)Moonshot API list price

Ceilings measured on Modal (serverless, per-second billing) at $56.79 per hour for the 8x B300 node, with request parallelism verified per load level and flat throughput across consecutive levels. The comparison holds at saturation only: at our benchmark's utilization the same box cost about 14x more than the API per token. Breakeven = saturated cost over API price: about 20% of ceiling for output-heavy work, 24% for fresh input.

Measured saturation ceilings against API list prices, per million tokens, log scale. Output and fresh input favor the box; cached input is the API's only win.

At saturation the box produces output tokens at $3.01 per million against the API's $15.00, and processes fresh input at $0.73 per million against $3.00. Cached input is the single direction the API wins, $0.30 against about $0.35. These ceilings come from untuned MoE kernels on a 2-day-old model, on a single node; tuned kernels, speculative decoding, and disaggregated serving would lower them further.

Those ceilings are priced at our rate, and rates vary widely by deployment type. We rent from Modal, serverless with per-second billing and scale-to-zero, at $7.10 per B300-hour. Tracked market rates for the same GPU in July 2026 average $8.69 on demand (Oracle lists $15.00), $6.33 reserved (down to $3.27 on a 36-month commitment), and $3.18 spot. Every dollar figure above scales linearly with that rate: at spot pricing the output ceiling drops to about $1.35 per million tokens, and even at the priciest hyperscaler on-demand rate it stays near $6.40, still under half the API price. Serverless carries a premium per hour but bills nothing when idle, which is what makes the boot-on-demand pattern below work; the usual breakeven against an always-on instance sits near 30% utilization.

The load-bearing word is saturation. Rent runs whether the GPUs work or idle, and at our benchmark's actual utilization the same hardware cost about 14x more than the API per token. The breakeven has a clean form: saturated cost divided by API price, as a share of the ceiling. For output-heavy work that means holding the box above about 20% of its measured ceiling, roughly 1,100 output tokens per second; for fresh-input work, about 24%, roughly 5,300 tokens per second. Cache-heavy reasoning work like our benchmark is the hard case: its comparator price sits near the box's own ceiling cost, so it breaks even only when nearly saturated, and with API cache discounts factored in it may never.

The speed numbers close the loop. Our saturated per-stream rate, 20 to 27 tokens per second, matches the API's everyday streaming speed almost exactly. A saturated batch server looks slow from the outside, and that operating point is what every per-token API sells. Running your own box is the only way to buy the fast lane.

Cost per run

Extraction cost vs recall, with K3 · N=3red flagsgreen flags

Extraction cost only (screener calls; the fixed judge overhead is excluded, so the baseline is $0). Faded points are the previous open generation. The dashed band is rent divided by throughput, which depends on utilization; the arrow marks $6.00 at the 38-minute window our 6 runs actually shared. K3 through the API costs $2.24 per run at 0.64 recall; GLM-5.2 delivers 0.68 for $0.34.

Extraction cost only (screener calls; the fixed judge overhead is excluded, so the baseline is $0). Faded points are the previous open generation. The dashed band is rent divided by throughput, which depends on utilization; the arrow marks $6.00 at the 38-minute window our 6 runs actually shared. The API arm ran on red flags only, so it has no K3 point here.

The June cost-recall chart with K3 added. The API point is a per-token price; the self-hosted band is rent divided by throughput at 2 utilization levels.

Through the API, K3 costs $2.24 per benchmark run. Reasoning tokens bill as output at $15.00 per million, which makes it the priciest open-weight contestant we have run, at recall below the closed trio.

A self-hosted run is priced differently: GPU rent divided by whatever you push through the box. Our 6 benchmark runs came to $6.00 each, and that number says nothing about the deployment: 6 concurrent runs are a tiny workload for a 512-slot machine, and a benchmark has no more work to offer it. Priced at the box's measured saturation throughput, the identical run costs about $1.40. Same hardware, same config, same calls; the only variable is how many neighbors share the rent. That 4x spread on our own workload is the utilization argument in miniature.

On this task, GLM-5.2 stays on the cost-recall frontier either way: 0.68 red-flag recall for $0.34.

What we ran

The benchmark is unchanged from our June eval of 8 models: extract red flags (warning signs) and green flags (positive signals) from real SEC 8-Ks, 23 and 22 filings, 3 runs each, through our production screener prompts. The score is pooled_coverage: the share of frozen reference flags the model found.

K3 joined after the references were frozen, so it never fed the consensus pool: real flags it finds that the pool missed count for nothing, and all its scores are floors. Rebuilding the reference around it would break comparability with the June table, so we audited the floor instead, asking the judge for material, filing-supported K3 findings the reference lacks. It found 5 across both tasks (2 late-filing notifications on filings the reference counts as clean, plus 3 variants of 1 takeover event); crediting all of them would move K3 by 0.02 to 0.04 and reorder nothing.

Quality: frontier-class, if you serve it right

We served the same weights 3 ways: on our box with a plain vLLM config, on our box with the vendor's recommended config, and through Moonshot's API as the reference point. Red-flag recall came back 0.42, 0.61, and 0.64, with the API arm's 0.64 itself a floor: it covers red flags only, ran through 1 provider (OpenRouter routing to Moonshot), and lost 1 to 4 of 84 calls per run to benign errors. On green flags, where the comparison is between the 2 self-hosted configs, the vendor config puts K3 in the top 3 of the June field, past GPT-5.5 (0.47) and Sonnet (0.41), behind Opus (0.62) and GLM-5.2 (0.57). Frontier-class intelligence, running on hardware you rent by the hour.

Kimi K3 pooled_coverage by serving setup · mean ±std, N=3self-hosted, no parserself-hosted, reasoning parserMoonshot API

Same weights (official MXFP4 checkpoint) on the same filings. Adding the vendor's reasoning parser to the vLLM serve command moves red-flag recall from 0.42 to 0.61 and green from 0.34 to 0.54. Dashed lines mark the June leaderboard context. The API arm ran on red flags only.

Same weights under 3 serving setups, with the June leaderboard as context. The reasoning parser accounts for the whole jump; the remaining 0.03 to the API sits within config and noise range.

The spread has 1 cause. Our pipeline requests JSON through response_format: json_object, and on vLLM that switches on grammar-constrained decoding: every generated token must continue a valid JSON document, so a reasoning model's first token has to be { and it never gets to think. The vendor config includes --reasoning-parser kimi_k3, which routes thinking to a separate channel and constrains only the final answer. vLLM's reasoning-outputs docs describe the split. Without the flag, K3 averaged about 180 output tokens per call. With it, about 1,500, and recall moved from 0.42 to 0.61 on red flags and 0.34 to 0.54 on green. The vendor config also switches the KV cache to fp8; the parser dominates the delta, though the 2 changes can't be separated exactly.

The flag itself is documented; the trap is that the failure is silent. The docs present the parser as output formatting, the model card's own vLLM snippet is a bare vllm serve with no flags at all, and no warning we could find says that JSON mode without the parser deletes thinking rather than leaving it mixed into the text. GitHub threads exist of users mapping this interaction by experiment; they lack a number for the quality cost. Now there is one: 0.19 recall on both of our tasks. If your pipeline requests structured output from a reasoning model, confirm the server separates thinking from the answer.

Making self-hosting pay

The ceilings only matter if your workload can reach them. Ours could not at benchmark time: 6 serial streams against a 512-slot server. Getting the economics requires shaping the work before renting the box.

Feed it from a queue, not from users. Breakeven needs the box above roughly a fifth of its ceiling around the clock. Backfills, re-screens, and batch extraction have that shape; interactive traffic rarely does.
Verify the concurrency is real. Our eval pipeline carried a concurrency setting of 50 and ran serially anyway: the sum of per-call durations equaled the wall clock. When those 2 numbers match, you have 1 stream, whatever the config says.
Group repeated context. vLLM's prefix cache makes a re-sent document nearly free, the same way API caching discounts it. Send all screeners for 1 filing together and share system prompts across calls.
Accept slower streams at saturation. Per-stream speed drops to 20 to 27 tokens per second when the box is full. Latency-sensitive traffic wants the opposite trade.
Make boots cheap, then boot on demand. --load-format fastsafetensors cut our 1.53 TB weight load from 31 minutes to 11, and the boot cost from about $37 to about $20. At that price, accumulating work into windows, booting, saturating for an hour, and tearing down beats keeping a warm idle box by a wide margin.
With reasoning on, plan for a tenth of the request throughput. Thinking tokens are decode work like any other output.

Where we sit

At Bollwerk Labs, we help companies transform how they detect and eliminate threats. If you're weighing owned weights against APIs for work like this, we're happy to compare notes: hello@bollwerk.ai.