Guided Decoding & Structured-Output Reliability

We took one decision task (a consultation-booking eligibility check), one model (Llama 3.1 8B Instruct), and one set of 25 customer messages with known-correct outcomes, and ran 50,000 measured generations across ten conditions: five output designs, each with and without grammar-enforced (guided) decoding via vLLM + xgrammar.

TL;DR

¹ Action, reason code, criteria flag, and appointment all correct. n = 5,000 per row (25 prompts × 200 runs) per enforcement condition; enforced decoding held schema validity at 100% in every design and never changed decision correctness by more than ±0.1pt.

Three takeaways, in order of practical importance:

1. Schemas should capture facts; code should apply policy. Every attempt to make the schema itself compute (designs 2–4) plateaued around 90% correct. Splitting the turn into LLM-extracts-facts → ~20 lines of Python decide → LLM-narrates hit 10,000/10,000 correct decisions — and used fewer tokens than any scratchpad design.
2. Enforced decoding buys validity and field order, not correctness. Decision accuracy was statistically identical with and without enforcement in every design. What enforcement bought: 100% parseable output where unenforced degraded to 97.9% on complex schemas (p ≈ 10⁻²¹), guaranteed field ordering, at ~10–20% throughput cost.
3. The baseline fails silently, confidently, and deterministically. On scenarios requiring arithmetic before the decision, the base schema booked the ineligible customer 400/400 times at temperature 0.7 — fluent, schema-valid, internally consistent wrong answers.

Full findings with per-scenario breakdowns: available on request. Want to replicate the results?: Contact us

The task

The model plays a booking assistant applying a fixed eligibility policy to a customer message (common.py, _POLICY):

• Customer must be aged 18–70 inclusive (under 18 → decline, reason under_age).
• Total requested service value must be ≤ £10,000. Totals of £10,001–£12,000 → offer_alternative (reduced-scope consultation); > £12,000 → decline; both with reason over_limit.
• Age or total undeterminable → escalate_human, reason insufficient_info.
• All criteria met → book_appointment with the requested date/time.

The policy text is byte-identical across every condition. Only the shape of what the model is asked to produce changes.

Test set

25 prompts (scenarios.json), five per scenario group, each with a known-correct expected outcome (plus 3 optional insufficient_info controls behind --include-controls):

The ten conditions

Two orthogonal factors:

• Enforcement: constrained (vLLM structured outputs, xgrammar backend — the grammar guarantees the schema and its property order) vs unconstrained (identical prompt describing the same format, nothing enforced). Switched per request:

# run_eval.py — the ONLY difference between enforced and unenforced
extra = {"seed": seed}
if enforced:
    extra["structured_outputs"] = {"json": SCHEMAS[variant]}

resp = await client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "system", "content": SYSTEM_PROMPTS[variant]},
              {"role": "user", "content": case["prompt"]}],
    temperature=0.7,
    max_tokens=400,
    extra_body=extra,
)

• Output design (schema variant): a progression that moves ever more "reasoning" into the structured output — and then abandons that idea entirely.

Design 1 — base schema, validators-first ordering

Decision fields are declared (and, under xgrammar, generated) before the free-text reply, so the model must commit to a decision before it can write the socially pleasant sentence:

SCHEMA = {
    "type": "object",
    "properties": {
        "eligibility": {
            "type": "object",
            "properties": {
                "criteria_met": {"type": "boolean"},
                "reason_code": {"type": "string",
                                "enum": ["eligible", "under_age", "over_limit", "insufficient_info"]},
            }, ...
        },
        "action": {"type": "string",
                   "enum": ["book_appointment", "decline", "offer_alternative", "escalate_human"]},
        "appointment": {"anyOf": [{...date/time...}, {"type": "null"}]},
        "response_text": {"type": "string", "maxLength": 700},   # generated LAST
    },
    "required": ["eligibility", "action", "appointment", "response_text"],
    "additionalProperties": False,
}

Designs 2–4 — making the schema "show its working"

Each adds fields ahead of the decision so the reasoning is forced onto the page before the model commits (compute-before-commit):

# Design 2 "reasoned": free-text arithmetic + an explicit band, before eligibility
SCHEMA_REASONED = {..., "properties": {
    "calculation": {"type": "string", "maxLength": 400},
    "within_limit_banding": {"enum": ["within_limit", "reduced_scope_band", "over_limit"]},
    **SCHEMA["properties"],   # then the decision fields, then response_text
}}

# Design 3 "tiered": unambiguous boolean gate + conditional tier
"exceeds_value_limit": {"type": "boolean"},
"over_limit_tier": {"anyOf": [{"enum": ["reduced_scope", "hard_decline"]}, {"type": "null"}]},

# Design 4 "table": every predicate the policy needs, as its own non-nullable field
"customer_age": {"type": "integer"},
"age_within_18_to_70": {"type": "boolean"},
"total_requested_value": {"type": "number"},
"total_at_most_10000": {"type": "boolean"},
"total_at_most_12000": {"type": "boolean"},

Design 5 — the pipeline: extract → code decides → narrate

The model never makes an eligibility decision. Call 1 extracts facts only (under enforcement it is grammatically forbidden from deciding or even summing — the extraction schema has no decision fields). Plain Python applies the policy. Call 2 writes response_text given the code's decision.

EXTRACTION_SCHEMA = {..., "properties": {
    "customer_age":     {"anyOf": [{"type": "integer"}, {"type": "null"}]},
    "price_components": {"type": "array", "items": {"type": "number"}, "maxItems": 10},
    "requested_date":   {"anyOf": [{"type": "string", "maxLength": 20}, {"type": "null"}]},
    "requested_time":   {"anyOf": [{"type": "string", "maxLength": 10}, {"type": "null"}]},
}}

def apply_policy(age, components, requested_date, requested_time):
    """The policy as code — ~20 lines, verified against all 28 expected outcomes
    independently of any model."""
    total = round(sum(components), 2) if components else None
    if age is None or total is None:
        return result(False, "insufficient_info", "escalate_human")
    if age < 18:
        return result(False, "under_age", "decline")
    ...
    if total > 12000:
        return result(False, "over_limit", "decline")
    if total > 10000:
        return result(False, "over_limit", "offer_alternative")
    ...
    return result(True, "eligible", "book_appointment", {"date": date, "time": time})

What we measured

Every call is logged as one JSONL record. All scoring is rule-based and auditable — no LLM judges anywhere (common.py):

• valid_json — parses and matches the schema (hand-rolled validator, validate_schema(), so the pass/fail rules are explicit in the repo). Parsing is lenient for the unconstrained condition (first balanced {...} block), so it isn't failed on markdown fences alone; strict-vs-lenient is logged separately.
• decision_correct / fully_correct — action matches the expected outcome / all four decision fields match, with date/time normalisation (14:30 == 2:30pm).
• response_consistent — does the free-text reply agree with the structured decision it shipped with? A regex checker flags, e.g., booking-confirmation language ("you're booked", "see you on", "confirmed for…") when action is decline:

def check_consistency(obj):
    flags = []
    if obj["action"] == "book_appointment":
        if obj["appointment"] is None:                 flags.append("book_but_appointment_null")
        if _any(_DECLINE_LANG, obj["response_text"]):  flags.append("book_but_decline_language")
    else:
        if obj["appointment"] is not None:             flags.append("non_book_but_appointment_populated")
        if _any(_BOOKING_CONFIRM, obj["response_text"]): flags.append("booking_confirmation")
        if _TIME_AS_AGREED.search(obj["response_text"]): flags.append("time_stated_as_agreed")
    ...

• field_order_ok — was response_text emitted after the decision fields in the raw text? (Trivially true when constrained; the unconstrained rate is the interesting number — it was 100%.)
• latency_ms, completion_tokens, finish_reason, peak_vram_mb, raw_output — cost/ops telemetry, plus the raw text for eyeballing failures.

Fairness controls in the runner (run_eval.py): seeds derived from (question_id, run_index) so both conditions see identical seed pools (paired comparison); ~10 discarded warmup calls per condition so one-time grammar compilation doesn't pollute latency; per-condition blocks with alternating order so neither condition systematically gets the warmer cache. Differences are tested with Fisher's exact test per scenario group (analyze.py).

Key findings

The baseline's 17-point gap is arithmetic, not attitude. It concentrates in the four scenarios where eligibility requires summing components before the threshold check — on those, criteria_met: true was emitted with zero variance across 400 runs. The schema places the decision token before any space to compute, so the model never computes. Notably, the failure mode this eval was originally designed to catch — response_text contradicting the structured decision — essentially never happened (<0.2%): whatever the model decides, it narrates faithfully.

Scratchpads fix only what they force onto the page. The calculation field made clear over-limit multi-part cases go 0% → 99.5% — but band-boundary cases collapsed: the model would correctly write "£6,500 plus £4,250 is £10,750" and then classify it into the harshest category anyway. Accuracy tracked distance above the threshold (£10,750 ≈ 0%, £11,250 ≈ 42%, £11,600 ≈ 86%).

A schema is not a computation graph. In the decision-table design, total_at_most_12000 matched total_at_most_10000 in 99.5% of 10,000 runs — including 97.3% of runs where they should differ. Trivially-true predicates were answered false on ineligible customers (the verdict contaminating an unrelated comparison), and downstream fields routinely ignored upstream fields (per-question agreement between the action and the model's own booleans fell as low as 8%). The predicate fields look like an audit trail; they are not one. One capability was near-perfect throughout: extraction (ages 100% correct in 10,000 runs).

The pipeline doesn't just score better — it fails differently. Every schema design's failure mode was a confident, fluent, wrong booking. The pipeline's one observed development-time failure (an empty extracted price list) routed to human escalation — it fails safe, and the residual risk (occasional numeric extraction slips) is guardable with more code: verify each extracted number appears verbatim in the source text.

Reproducing

Interested in re-running or iterating on our experiment? Contact our team and we'll share the runbook.

Requirements: a local vLLM server (v0.22.x, xgrammar structured-output backend), openai, scipy, optionally nvidia-ml-py. Model weights in the local HF cache.

# 1. serve (single server for all conditions; enforcement is per-request)
./serve_vllm.sh

# 2. sanity check (16 calls, ~1 min)
python3 run_eval.py --smoke

# 3. full runs — one pair of conditions at a time (~45–60 min each at concurrency 32)
python3 run_eval.py --n 200 --concurrency 32 --out results.jsonl                          # base pair
python3 run_eval.py --n 200 --concurrency 32 --conditions reasoned --out results_reasoned.jsonl
python3 run_eval.py --n 200 --concurrency 32 --conditions tiered   --out results_tiered.jsonl
python3 run_eval.py --n 200 --concurrency 32 --conditions table    --out results_table.jsonl
python3 run_eval.py --n 200 --concurrency 32 --conditions pipeline --out results_pipeline.jsonl

# 4. analyse everything together (rates, Fisher's exact tests, latency mean/p95)
python3 analyze.py --results results*.jsonl --csv summary_all.csv

Machine-specific caveats: Unified-memory VRAM reporting was ommited in our test, latency under batching (which are arguably a better metric for most commercial set ups) were tracked and recorded.

Repo layout

Limitations

• One model, one size. This study used Llama 3.1 8B Instruct. Larger or reasoning-tuned models may improve the absolute schema-only results and shift where performance plateaus. We suspect the underlying failure mechanisms observed here—such as predicate copying and verdict contamination—may persist, but that has not been tested and should not be treated as a model-independent finding. Llama 3.1 supports a context window of up to 128,000 tokens; our deployment was configured for 30,000. Design 4 used only 166 tokens, so the experiment did not test behaviour under context-window pressure.
• Rule-based consistency checking has edges; flagged raw_output was spot-checked by hand, but the ~0.1–0.8% inconsistency rates should be read with that in mind.
• Latency figures are client-observed under concurrency 32 (batched serving) — fair between conditions, not single-user absolute numbers.
• The policy is deliberately small. That's the point — even a policy this simple defeated every schema-as-computation design — but real policies have more branches, and the pipeline's advantage should grow, not shrink, with policy complexity.