Tara is a LoRA fine-tune (rank 16, alpha 32) of Llama 3.1 8B Instruct, built to guide a mortgage lead through a compliant, structured conversation to a booked adviser appointment. Loss curves tell you the model got better; they don't tell you why. Because the fine-tune is a LoRA adapter, every adapted matrix was shifted by exactly ΔW = (α/r)·B·A, and that shift can be measured directly and compared to the base model's own weights — recovered as merged − ΔW, so no separate base checkpoint download was needed. No prompting, no LLM judge: everything below is measured straight from the 224 adapted matrices (7 projections × 32 layers, 6.98B parameters touched).
TL;DR
| Measurement | Value | Reading |
|---|---|---|
| Schedule-action match, held-out scenarios | 52% → 91% | The behavioural gain the weight analysis explains |
| Network-wide relative shift ‖ΔW‖F / ‖W_base‖F | 2.68% | Small, targeted parametric change |
| Largest single-matrix shift | 6.2% | down_proj, layer 1 — input-encoding outlier |
| Update energy in top 1 / top 4 of 16 directions | 41% / 73% | Effective rank of the learned change ≈ 3–4 |
| Adapter drift, checkpoint 700 → 768 | <0.05% | Training had genuinely converged (30.332 → 30.346) |
Three takeaways, in order of practical importance:
- The gain is concentrated where routing decisions live. gate_proj — the mechanism that decides how strongly each MLP channel fires — moved most (3.30% mean shift, peaking through layers 22–28). k_proj moved least (1.50%): what the model attends to barely changed; what it does with that attention changed considerably more.
- The learned change is effectively rank 3–4, not a rewrite. Each matrix had 16 available update directions under the rank-16 adapter; 41% of total update energy sits in one direction and the top four account for 73%. A handful of coherent behavioural adjustments layered onto an otherwise intact base model.
- The validators-first schema (v2) beat v1 throughout training, not just at the end. Lower loss and perplexity at every step — an architectural effect of autoregressive conditioning, not a data-volume effect.
Training data, evaluation set and per-scenario results: available on request.
Rationale
General-purpose models are built to answer millions of different questions reasonably well. Tara is built to do one thing exceptionally well, and that difference in objective is the whole argument for fine-tuning rather than prompting a frontier model and hoping the guardrails hold. Moving primary generation onto our own fine-tuned model gives us five things a wrapped API call can't:
- greater behavioural consistency across thousands of concurrent conversations,
- tighter regulatory control over what the model will and won't say,
- complete ownership of the improvement cycle,
- materially lower inference cost at scale, and
- independence from any single model vendor's roadmap or pricing changes.
It also closes a loop a prompt-only system can't close at all. Call-Bridge scores every call and conversation thread for quality, sentiment and outcome, and that structured data feeds directly into the same pipeline that trains our models. Real conversations improve the model; the improved model produces better conversations. The compounding loop is the proprietary asset — not the model weights on their own, but the pipeline that keeps improving them.
Training
Tara is on its second schema iteration: v1 used our original data schema, v2 restructures the same underlying conversations around a validators-first output format.
The reasoning behind v2 is architectural, not organisational. LLMs generate autoregressively — each token is conditioned on every token generated before it in the sequence. By restructuring the output so the guardrail and validator tokens are generated first, the response tokens and the final action tokens are both generated conditional on the validation step, rather than being produced independently and checked after the fact. Validation isn't a filter bolted on afterwards; it's part of the context the model is reasoning from when it produces the response and decides the action. That is the structural reason v2 tracks lower loss and perplexity than v1 throughout training — not a data or schema-tidiness effect.
Both runs converge cleanly — loss and perplexity fall sharply in the first ~50 steps, then settle into a smooth decline across three epochs with no sign of instability. The learning-rate schedule is a short warmup to a peak around 2e-4 followed by cosine decay to zero by the final step: standard practice, included so the shape of the run is fully inspectable rather than taken on trust. Because v2 leads v1 at every point in training, the gain came from the schema redesign itself, not simply from more steps or more data.
Weight analysis
Every adapted matrix's shift ΔW = (α/r)·B·A was computed exactly and normalised against the base weights, giving a per-matrix relative shift ‖ΔW‖F / ‖W_base‖F. Aggregated by projection type:
| Projection | Mean relative shift | Role |
|---|---|---|
gate_proj | 3.30% | Decides how strongly each MLP channel fires — moved most |
up_proj | 2.76% | MLP input expansion |
v_proj | 2.65% | Attention value projection |
o_proj | 2.64% | Attention output projection |
q_proj | 2.03% | Attention query projection |
down_proj | 1.96% | MLP output projection |
k_proj | 1.50% | Attention key projection — moved least |
gate_proj's dominance, peaking through layers 22–28, is the signature of learned routing and decision behaviour: when to propose a time slot, when to ask for confirmation, when to escalate. k_proj barely moving means the attention pattern itself — what the model looks at — survived fine-tuning nearly intact. Two early-layer outliers, down_proj in layer 1 (6.2%) and v_proj in layer 0 (4.2%), sit where the model adapts to Tara's specific prompt and output structure, reshaping how the task format is encoded on the way in.
The update is concentrated, not diffuse. Each matrix's adjustment has 16 available directions under the rank-16 adapter, but 41% of the total update energy sits in a single direction and the top four account for 73% — the learned change is effectively rank 3 to 4. And it had genuinely finished: between checkpoints 700 and 768 the total update norm moved by under 0.05% (30.332 → 30.346), confirming the adapter had stopped drifting well before the run ended.
The weight-space counterpart of the eval result: a large behavioural gain produced by a small, targeted and fully inspectable parametric change — not a black box.
Benchmarks
Independent of the weight analysis, Tara's outputs were scored against a 1,000+ scenario evaluation set covering conversational quality, workflow accuracy and regulatory compliance, alongside the base 8B model, Claude Sonnet, Claude Haiku, GPT-3.5 and GPT-4 Turbo. The eval dataset was not exposed during training — the merged model saw it for the first time at eval.
Tara's behavioural consistency — how tightly its lower-quartile responses cluster — beats Claude Sonnet, Claude Haiku and GPT-3.5. Only GPT-4 Turbo scores marginally higher on pure structural compliance. On LLM-judged conversational quality, Tara performs comparably to Claude Sonnet while being noticeably more consistent, and outperforms both Haiku and the OpenAI models.
The headline point: every other model in the comparison either follows the rules or sounds natural. Tara does both, on a model roughly 8× smaller than Sonnet or GPT-4 Turbo — materially faster and cheaper inference, with real headroom as the model scales.
Implications
- Vendor independence: we control our own roadmap, deployment schedule and operating costs, rather than inheriting changes made by an external model provider.
- Data governance: customer conversations are not shared with external AI providers for inference or training on the primary path; Tara is hosted within our own enterprise cloud infrastructure, supporting a GDPR-first architecture with full control over data processing and retention.
- Compliance by design: fine-tuned specifically for regulated financial-services interactions, with structured outputs and deterministic workflows that support continuous evaluation against Consumer Duty and FCA standards.
- A compounding asset: Call-Bridge feeds real conversation outcomes back into the same training pipeline that produced these results. For customers with sufficient scale, this also lends itself to customised adapters and pipelines to further improve conversions.
Limitations
- Scope of the claim. These figures describe a targeted behavioural improvement on the defined mortgage-engagement task. This is not a claim of general model superiority — evaluation scope and methodology should be read with the result.
- Interpretation is signature-based, not causal. The projection-level pattern (gate_proj up, k_proj flat) is consistent with learned routing behaviour, but weight movement tells you where the change concentrated, not a verified causal account of each behaviour.
- Judge scores are LLM-judged. Figs 2–3 inherit the biases of their judge models; the behaviour score (Fig 4) is the structural, rule-based measure and should be weighted accordingly.