docs(specs): model routing analytics design

[Copilot]

Design proposal for a "Model routing" analytics page that reads only unified session files (usage/aw_session.jsonl). It covers views, exact metric definitions with the event and field each reads, and a proposed runner- or proxy-written record for each data gap. This PR adds only docs/src/content/docs/specs/model-routing-analytics.md. It does not change the page, the collector, AWF, or other specs.

Reader rules (MRA-nnn)

• Trust order: proxy records (firewall.*, model.inventory) come first, then workflow.info. model_routing.outcome is advisory and never decides state, pick, or attribution.
  • This matters because workflow.info.modelRouting.status can disagree with the proxy. The copilot-budget-exhausted-steering fixture reports rejected/harness_selection_mismatch for a run the proxy recorded as selected and then budget-failed.
• Older files: camelCase and snake_case field names are both accepted. Legacy ["endpoint"] deviations are normalized below model-routing/v0.28.39. Unknown events and fields are ignored.
• Unknowns:
  • A missing value stays unknown, never zero or success.
  • Every sum carries a known/unknown count.
  • Partial runs are excluded from medians and comparisons.
  • Overlapping totals are never added: firewall.token_usage with usage.report, cumulative totalAic, and agentMetrics snapshots.
• Privacy: no message, detail, error, or reason text; only identifier-shaped values are shown.

Run model

• Routing state: selected → selection_failed → harness_reported → unrouted / configured_unrecorded (from proposed model.config) → unrouted_inferred → unknown. Proxy evidence wins over configuration; a disagreement is flagged.
• Request attribution: each request goes to exactly one of eight groups: detection (by phase, not purpose), evals, classifier, sub-agent exact, sub-agent correlated, main agent, agent not split, unattributed. Cost is complete only when the accounting source is present, every record has aic, and every request record has an accounting record.
• Deviation causes: sub-agent fallback, sub-agent model, harness substitution, effort change, endpoint change, unexplained.

Views

Adoption, decisions (pick stability, including within each classifier label cell), cost (main agent, sub-agent by definition, classifier, detection, routed vs. unrouted), deviations, failures (exit 78, budget, steering, upstream errors, unavailable models), outcomes (success rate and cost per success by pick within a label cell), and the single-run drill-down. A table shows what each view can show on current data.

Data gaps

Gap	Finding	Proposal
G-1 Model config	Goal/mode exist only on selection records; allowed and blocked models and the credit budget are absent	model.config from aw_info.json
G-2 Candidate pool	AWF writes ranked_choices etc.; the collector drops them. Predictions don't exist at runtime (tables are ordered identities only)	Keep the fields; routing-table v2 with predicted_success / predicted_aic / excluded_choices
G-3 Router/tables/classifier	router.version is the package version; table identity is missing; classification records have no requestId	router.image, router.tables.*, classification requestId
G-4 Run outcome	evaluate_outcomes.cjs results never reach a session; pending items are never re-checked	safe_output.outcome; per-item re-evaluation
G-5 Upstream errors	Already ingested as a generic firewall.event, but the error code is lost	firewall.upstream_error with identifier-only errorCode / errorType / errorParam
G-6 Baseline price	Missing	model.baseline_estimate repriced via computeInferenceAIC
G-7 Inventory dropped	Wider than reported: any routing_models: null rejects the whole snapshot, and models: null → [] loses configured	Collector fix
G-8 Detection	No gap: identified by phase detection	None
G-9 Harness	Copilot CLI and Copilot SDK can't be told apart	workflow.info.engineHarness
G-10 Request ownership	No caller identity on proxy records	AWF records an x-awf-caller header as purpose / agentName, labelled caller-declared
G-11 Dropped request fields	status, unavailable, and endpoint fields are dropped by the projection	Keep them
G-12 Run identity	No attempt, workflow ref, or server URL	Add to workflow.info

For reviewers

• The unified session spec has no related-documents section (closest is §11.2), so it is not linked or edited.
• Open question: does AWF guarantee that a rejected request incurs no AIC? Until confirmed, such runs are treated as partial cost.

[SivaKesava1]

Thanks, this is a solid design. A few corrections from the routing side:

1. G-3, router identity. The router image default has moved on: since gh-aw-firewall#10104 the router is published with every AWF release as ghcr.io/github/gh-aw-firewall/router:<awf-version>, gh-aw switches its default to that image next, and githubnext/gh-aw-router is being retired. So router identity is the AWF version plus the table identity. router.tables.* (schema version, profile, digest) is still needed because tables can be mounted separately; router.image can stay for the override case (sandbox.agent.images.router).
2. G-2, candidates. Please keep step 1 (retain rankedChoices, eligibleChoices, catalogueOverlap) and drop step 2. The page should not record or show per-candidate predicted success or predicted credits: the routing tables are ordered lists by design, and the estimates behind that order are not calibrated enough to display next to observed results. The page should show the ranked list and what was actually picked and observed. This also answers open question 3.
3. Open question 6, rejected requests. Yes for AWF's guard rejections: rate-limit and model-policy blocks mark the request rejected before it is forwarded (proxy-guards.js, rate-limit.js), so no provider sees it and no AIC is charged. One exception to model: AWF's own reasoning-effort check (validateReasoningEffort in containers/api-proxy/upstream-http.js) also answers locally with a 400 unsupported_reasoning_effort and never reaches the provider, but it is recorded as failed, not rejected. Requests failed with that code are also zero-cost and should be grouped as AWF-local failures.

[SivaKesava1]

@rabeckett could you review this design for the model routing analytics page? It can't add you as a requested reviewer (read access), so tagging you here. The routing-side corrections are in the comment above.

[pelikhan]

I think this probably goes in cao

[Copilot]

Thanks, this is a solid design. A few corrections from the routing side:

1. G-3, router identity. The router image default has moved on: since gh-aw-firewall#10104 the router is published with ev...

Done in e72dfaa (0.2.0, doc only):

• G-3 and the Section 4.2 dimension now treat router identity as the AWF version plus the table identity. router.tables.* and router.image (for the sandbox.agent.images.router override) are kept. router.version is used only to label runs on the retired githubnext/gh-aw-router image.
• G-2 keeps only the collector change and states that the page shows no per-candidate predictions. I removed predictions from the views, the rollout, and the gap summary, and resolved open question 3.
• New rule MRA-073: a rejected request record and a failed unsupported_reasoning_effort request record both count as zero cost. Section 5.3 completeness uses this rule, the failures view has a separate "AWF-local failures" group, and open question 6 is resolved.

Gap you should know about: request records don't carry an error code today. The code only appears on the run's failure record, which has no requestId. So with current files, a failed request without an accounting record is treated as an AWF-local failure only when it is the run's only such request and the run's failure code is unsupported_reasoning_effort. Otherwise its cost stays unknown. To fix this properly, I extended G-5 so AWF also writes error_code on failed request records.