Choosing the Right LLM Is an Engineering Problem, Not a Leaderboard

Summary

We tested nine out of the box models from our production stack to answer a question we get asked a lot: how much does parameter count actually matter? The short answer is far less than most people think, and not in the way most people assume. Past a fairly low floor, more parameters generally buy world knowledge and fluency, not reasoning capability, and on a structured decision task, that's the wrong thing to be paying for.

TL;DR

• Size predicts a floor, not a ceiling. Below ~8B dense parameters, models can't reliably hold a schema together. Above that floor, scaling further bought us almost nothing on this task: llama3.1 8B and llama3.3 70B (a 9x parameter gap, same family) scored identically on our two hardest scenario categories.
• Two models tied for best accuracy, on very different budgets. gpt-oss-120b and qwen3 30B-A3B both hit 100% decision accuracy. One needed 64GB of VRAM and 1.8 seconds to first visible token; the other needed 19GB and 12.5 seconds. Neither number is "better" in the abstract, it depends entirely on what you're building.
• Parameters buy world knowledge, not reasoning. The clearest evidence: gpt-oss-120b carries 6x the total parameters of gpt-oss-20b for a near identical active parameter budget, and the accuracy gap between them is within noise. The extra size shows up in fluency and tone, not in getting the rules right.
• This testing formed part of a broader range of eval work for a specific product we're developing. We shipped gpt-oss-120b for this specific deployment because of latency and tone, not because it was the most accurate or the cheapest model tested, it was neither. For a different job (a backend extractor with no one waiting on the response), the data points somewhere else entirely.

Explicit scope callout

Before the results, it's worth being precise about what this note does and doesn't cover.

Our model selection process is driven by several factors, and this note represents only one of them - out of the box performance on a basic arithmetic and reasoning task. Structured decoding (which we cover in a companion piece), response quality rubrics for tone and fluency, and fine-tuning all play a real part in how we actually choose and deploy a model. None of that is applied here.

These numbers are base model performance only. No structured decoding, no model specific prompt tuning. The prompt and question set is our internal standard reasoning/arithmetic/rule adherence suite, the same one used in our structured decoding piece. Our experience is that out of the box performance is still a good indicator of what's achievable once the inference engine is configured properly, the prompt is tuned, and, where needed, the model is fine-tuned. It is a starting point, not a final verdict on any model's ceiling.

The test scenario

The rig asks each model to reason over customer appointment-booking requests against a fixed rule set:

• The customer must be aged 18 to 70 inclusive.
• The total value of the services requested must be £10,000 or less.
• If both criteria are satisfied, book the customer's requested appointment.
• If the customer is under 18, decline with reason_code: "under_age".
• If the total exceeds £10,000:
  • For totals between £10,001 and £12,000 inclusive, offer a reduced-scope consultation using action: "offer_alternative".
  • For totals above £12,000, decline using action: "decline".
  • In both cases, use reason_code: "over_limit".
• If the customer's age or total requested value cannot be determined, use reason_code: "insufficient_info" and action: "escalate_human".
• Only populate appointment when action is book_appointment.
• Do not claim an appointment has been booked unless action is book_appointment and the appointment field contains the matching date and time.
• When declining, escalating, or offering an alternative, respond politely without implying the requested appointment is confirmed.

Each model was run across six scenario groups: simple_eligibility, simple_ineligibility, multi_step_eligibility, distractor_laden, tone_mismatched_or_adversarial, and insufficient_information_control_cases.

What we were testing

The experiment axis is the model itself: size, dense vs MoE, and family. If you're unfamiliar with Mixture of Experts (MoE): a dense model uses all of its parameters to produce every output token, whereas an MoE model selects which parameters to fire per token based on the input. gpt-oss-120b, for example, carries 116.8B parameters in total but activates only the 5.1B best suited to the task at hand.

The questions this design answers:

• To what extent does model size (active and total parameters) determine accuracy?
• Does an MoE model reason in line with its active parameter count, or its total?
• How much does model family affect performance on this task, independent of size?
• Since this is being evaluated for commercial deployment, we also measured memory and latency, both of which matter more or less depending on the specific application, which is a theme we return to throughout.

Models tested

We tested models already in our production stack, capped at 120B parameters (anything larger was excluded on cost grounds). At roughly 700 input tokens per call in this suite, we would be extremely unlikely to deploy something like a 1.2 trillion parameter model for this task, at roughly 32x the runtime cost of a 120B model for no meaningful gain we could justify testing for.

A note on two reasoning models

gpt-oss was trained from the ground up as a reasoning model, generating a response in channels:

1. an analysis channel that functions as chain of thought, followed by
2. a final channel that produces the visible output.

OpenAI ships reasoning_effort (low/medium/high) but no "off" switch, so gpt-oss benefits from CoT by design, on every call.

Qwen3 does have a documented reasoning off switch (a /no_think soft prompt token, from its hybrid training). In our local deployment (Ollama 0.30.5), the serving template pre-fills the <think> block before that switch can take effect. We spotted this when we saw no difference in TTFT, first content time, total time, or completion tokens with the soft switch applied, or with think: false / reasoning_effort: none. The model is already mid thought by the time either flag is caught.

The safe assumption, then, is that both gpt-oss and Qwen3's accuracy numbers include a chain of thought boost the dense Llama and Mistral models don't get. Whether that's an unfair advantage or a legitimate architectural strength is a matter of opinion. We lean toward the latter, since CoT is exactly how we'd serve these models in production. It's worth keeping in mind throughout the results below, particularly wherever a MoE/reasoning model is compared directly against a dense one.

Results

Does size predict reliability?

Put simply, no past a fairly low floor.

There is, however, a real floor. The gap between llama3.2 3B and llama3.1 8B is stark: the 3B model failed to produce valid JSON on 89 of 100 runs, and even when it did, only 7% of its decisions were correct. The 8B model (five billion more parameters and otherwise the same family) produced valid JSON 100% of the time and got 81% of decisions right, with a clean 100% on both simple scenario groups. Below a certain size, a dense model simply cannot hold the schema and the rule logic together reliably. That much of "size matters" is true.

What isn't true is that it keeps mattering once you're past that floor. llama3.1 8B and llama3.3 70B (same family, a 9x parameter gap, roughly 8x the memory footprint) land at 81% and 86% decision accuracy respectively, a difference well within what you'd expect from noise at this scenario mix. More strikingly, the two models scored identically on the hardest categories: both hit exactly 60% on multi_step and exactly 60% on adversarial. Nine times the parameters did not move the needle at all on the two scenario types where accuracy actually matters most. Whatever is holding the llama3 family back on multi step and adversarial reasoning, it isn't parameter count, it's something in the architecture or training that scale alone doesn't fix.

It's also worth separating two different kinds of failure that "decision accuracy" can hide. For llama3.2 3B and llama4 Scout, valid JSON rate and decision correct rate are nearly identical. These models mostly fail by never producing usable output at all, a reliability floor rather than a reasoning gap. For llama3.1 8B, Mistral Small 24B, and llama3.3 70B, the picture is different: all three produce valid JSON almost every time, yet still get the rule logic wrong 14–28% of calls. That's a genuine reasoning error, not a formatting one, and no amount of additional scale (see the 8B/70B pair above) appears to close it.

gpt-oss-20b is the standout on this axis. It outperformed every model except the two 100% scorers noted above (both of which carry the CoT advantage described earlier), slipping only on multi step and adversarial cases. In production terms that's a meaningful saving: against llama3.3 70B, gpt-oss-20b completed its run in roughly half the time, and at 12.85GB would fit comfortably on a single Nvidia T4 ($0.59/hr at managed hosting cost) against llama3.3 70B's requirement for a full A100 ($2.50/hr managed hosting) - around a sixth of the inference cost, for better accuracy on most scenario groups. We'd expect structured decoding plus a small extraction/validation pass ahead of gpt-oss-20b (roughly 20 lines of Python, covered in our structured decoding note) to close most of its remaining failures.

What is the real benefit, and downside, of MoE?

The two front runners, gpt-oss-120b (116.8B total / 5.1B active) and Qwen3 30B-A3B (30.5B total / 3.3B active), both handled 100% of scenarios correctly. On accuracy alone, it's a dead heat.

Time to first visible content is where the tie breaks decisively. gpt-oss-120b produces its first visible token in 1.8 seconds; Qwen3 30B-A3B takes 12.5 seconds, with total response time running to nearly 14 seconds against gpt-oss-120b's under 4 second total.

And therein lies our earlier explicit scope callout and note on why model fit for model use is relevant. In an asynchronous application (SMS for example, where total response time matters less) the sluggish response might be worth bearing for the lower memory overhead. However, for a synchronous application, such as a chat surface, that's not a performance gap, it's a different product (12+ seconds of silence before any output reads as broken, not slow). qwen3_30b_a3b accuracy is real, but it's time is spent almost entirely on reasoning tokens the user never sees, and no amount of correct but invisible reasoning helps a person who's already assumed the app has hung.

Nor is the memory saving as clean as it first looks once you price it properly. Qwen3 30B-A3B at 19.37GB runs comfortably on a single Nvidia A10 ($1.10/hr); gpt-oss-120b's 64.45GB needs a full A100 ($2.50/hr) to complete this job. On an idle inclusive hourly basis, Qwen3 looks like 44% of the cost. But if you're paying per inference rather than for idle server time, the picture flips: Qwen3's far longer generation window means its actual cost per response comes out at roughly 154% of gpt-oss-120b's. The "cheaper" model is only cheaper if the server sits mostly idle between calls; under load, the extra 10 seconds of generation time per call eats the VRAM saving and then some.

We could equally frame the above, considering a warm container, in potential queues or concurrency costs. Either way, the "cheaper" argument becomes less clear cut.

One last caveat before drawing any conclusion from a 100% score: it doesn't mean we'd deploy either model as is. Pre and post-hoc validation would still be needed to catch the roughly 1 in 1,000 plus errors that inevitably show up in production at scale. A 100% result on this scenario count is a strong signal, not a guarantee!

Why we opted for gpt-oss-120b - performance for the use case

Given the tie above, it would be easy to stop at cost and pick Qwen3 30B-A3B. We didn't, because this model sits directly inside a B2B customer-facing conversational flow, where it isn't just deciding, it's talking, turn by turn, with someone waiting on the other end.

For that surface, 1.8 seconds to first visible content versus 12.5 seconds isn't a latency difference, it's the line between a responsive assistant and one that reads as broken. Layered on top of that is a qualitative factor no accuracy metric captures: gpt-oss's tone and fluency out of the box matched what we wanted for this particular surface. That qualitative edge only mattered because gpt-oss had already cleared the bar on everything we could measure. It wasn't a trade against accuracy, since the two models tied on that. It was a decision to spend more on deploying a model fit for the surface its being deployed into. In fact, as an aside, we deployed our production on a single H100 to open up the full token limit.

So gpt-oss-120b didn't win this deployment because it was the most accurate model tested (it tied), and it isn't the cheapest (it's the most memory hungry model in the entire grid). It won because, for a synchronous, customer facing surface, it was the only 100% accuracy option that also felt instant and read well. Both of those were hard requirements here, not nice to haves.

What we gave up: memory is a real cash cost

None of the above is free. gpt-oss-120b's 64.45GB serving footprint is the largest of any model in the test - roughly 3.3x Qwen3 30B-A3B's 19.37GB for an identical accuracy result, and over 10x llama3.1 8B's 5.93GB. That's not an abstract number: it's the difference between fitting on a single mid tier card and needing a top end or multi GPU setup, and the gap multiplies with every concurrent worker in production. Read plainly: we paid a real, ongoing infrastructure premium for latency and tone on this surface, not a free win.

If we were deploying this as a backend extractor or orchestrator instead

Strip away the customer facing constraint and the calculus changes completely. A backend extraction or orchestration step doesn't need to feel instant to a person, it needs to be accurate and cheap to run at whatever concurrency the pipeline demands. Latency still matters, but as a throughput and SLA budget across a queue, not as a turn taking experience; a few extra seconds per call is often irrelevant if nothing downstream is blocked on it.

Under that framing, the decision becomes a straightforward speed/cost/accuracy trade, and the winner changes:

For a step with no user in the loop, Qwen3 30B-A3B becomes the strongest candidate: it matches gpt-oss-120b's accuracy at roughly a third of the VRAM, and the one thing that ruled it out for the customer facing case (12 seconds to content) stops being disqualifying the moment nothing is waiting on it in real time. gpt-oss-20b is the fallback if any latency SLA survives into the backend context, since it sits within noise of the top accuracy tier at a fraction of gpt-oss-120b's memory and still responds in about a second.

The point worth stating plainly: gpt-oss-120b isn't "the best model" in any absolute sense, it's the best model for a specific, latency sensitive, tone sensitive, doesn't need to understand world history, customer surface. The same benchmark data points to a different answer for a different job, which is precisely the argument for testing per use case rather than adopting a single default model across a stack.

What isn't baked into this note, but is potentially relevant

Everything above uses base models, pre-fine-tune, on a task that requires some arithmetic and rule-following. Some models (llama4 Scout among them) did badly on that specific axis. That doesn't make it a bad model; it makes it a good illustration of matching the model to the task. llama4 Scout, Mistral Small 24B, and gpt-oss-20b are among the most fluent, natural, and warm language models we use, by our own tone preferences. None of that shows up anywhere in a decision-accuracy number.

By contrast, gpt-oss-120b is strong for a verbose, considered response but not for getting to the point quickly. Qwen3 30B-A3B is fine on tone but prone to overthinking a task and introducing information nobody asked for (good for working something through out loud, less good at simply doing as instructed).

The takeaway: in a multi agent system, you generally want multiple model sizes, types, and families in the mix, each doing the part of the job it's actually suited to, not one model doing everything because it topped a single leaderboard.

Headline — speed vs memory vs accuracy

* Consistency rate is over schema-valid rows only; all other rates are over every attempted call. Mem GB is Ollama's reported allocation (weights + KV cache + graph) — the figure that sets the required card class, i.e. serving cost — falling back to peak system-memory delta where /api/ps was unavailable (NVML reports N/A on unified-memory boards).

Decision accuracy by scenario group

Speed detail (medians, serial single-stream)

Roofline = bandwidth ÷ active-weight bytes (273GB/s; active bytes = size_GB × active_B/total_B). A ratio well under 1.0 indicates overhead beyond weight streaming (attention/KV, kernel efficiency); reasoning models also spend tokens never seen in response_text, which further lowers this ratio without reflecting a real generation-speed problem.

Appendix: hardware and method

Hardware. Tests ran locally on our own Nvidia hardware; the first three calls to each model were excluded from results to absorb weight-loading/swap cost. Total available VRAM during the runs was 128GB, on a GB10 Grace Blackwell Superchip (Blackwell GPU with a 20-core ARM CPU at up to 1,000 AI TOPS, or 1 petaFLOP of sparse FP4 performance).

Method.

• Prompt-only structured output (guided decoding's unconstrained base condition — schema described in the system prompt, nothing enforced) — accuracy decomposes into rule-based layers with no LLM judge; every output is scored deterministically: valid_json → decision_correct → fully_correct → response_consistent.
• Serial requests, one model resident at a time. Concurrency 1 is load-bearing — batching would contaminate single-stream TTFT/decode numbers.
• Streaming metrics per call: ttft_ms (first token, including reasoning), ttfc_ms (first visible token — the user-perceived figure), decode_tps over the generation window, completion_tokens, total_ms.
• Three discarded warm-up calls per model absorb weight-load/swap cost.
• Memory (= serving cost): NVML reports N/A on our unified-memory hardware, so each model block instead records Ollama's own allocation (/api/ps size — weights + KV cache + graph, the figure that determines required card class) plus polled /proc/meminfo average/peak deltas against a clean baseline, with the resident model explicitly evicted before each block. One summary row per model in memory.jsonl.
• Reasoning: the grid measures as-served defaults. gpt-oss runs at reasoning_effort: low; Qwen3 thinks at its default — on this Ollama build (0.30.5) it thinks regardless of the /no_think soft switch, think: false, or reasoning_effort: none, none of which measurably changed TTFT, first-content time, total time, or completion tokens (the model is already mid-thought by the time any of these are caught). Reasoning tokens count toward decode_tps (they are generated tokens) but delay ttfc_ms; completion_tokens − visible_chunks ≈ reasoning spend per call, so this cost is reported, not hidden. The corollary, restated: Qwen3 and gpt-oss accuracy figures include a chain-of-thought boost the dense Llama and Mistral models don't get — a property of how we'd actually serve them, not an artefact of the test.
• Context pinned to 8,192 tokens across the grid via derived -8k model variants — Ollama otherwise auto-sizes context per model (131k for Llama 3.1, 40k for Qwen3, etc.), which would make KV-cache allocation, and therefore the memory metric, incomparable across models.
• Sampling: temperature 0.7, max_tokens 768 (2,048 for gpt-oss, since reasoning shares the same budget), seeds derived from (question, rep) so every model sees an identical seed pool.