Audit

The Audit page is the super-admin’s window into what’s been happening on your customer. Pick a user, pick one of their sessions, and you see exactly what was asked, what was answered, and every backend step the platform took to produce that answer — with sensitive data masked by default.

It only appears in the sidebar for the Customer Super Admin. Everyone else, regardless of role flags, has no access to it. The backend enforces the same restriction independently of the UI.

What you see

The page is laid out as four panels, drilling left-to-right:

┌────────┬──────────┬─────────────────┬──────────────────┐
│        │          │                 │                  │
│  USERS │ SESSIONS │  CONVERSATION   │  MCP CALL TREE   │
│        │          │                 │                  │
└────────┴──────────┴─────────────────┴──────────────────┘

Panel	What it shows
Users	The first 20 users in your customer (super admins + product app users). Includes a search box.
Sessions	The picked user’s recent chat sessions. Top 20, last 30 days by default, with a date range filter and a search box.
Conversation	The chat transcript for the picked session — every user prompt and assistant reply, in order.
MCP call tree	The full backend call graph behind every inference in this session. Each accordion row is one inference run; expanding it reveals the activities (MCP tools, AI calls) with their input, output, duration, and status.

Each subsequent panel shows a small loading overlay while its data is being fetched, so you always know which slice is in flight.

Refresh

The Refresh button (top-right) resets the page to step zero — clears your selections and re-fetches the user list. Useful when new sessions have been created since you opened the page, or when you want to step out of a deep dive and start over.

Locking personal information

Next to Refresh is the Lock Personal Information toggle. It controls whether the server masks PII before the data reaches your browser.

• Default: ON. Emails, phone numbers, credit cards, IBANs, IP addresses, US SSNs, UK NHS / NINO / postcodes, AU TFN / ABN / ACN / Medicare numbers, SA 13-digit IDs, EU VAT numbers, customer-id-shaped tokens, currency amounts (USD / GBP / EUR / AUD / ZAR + adjacent), percentages, dates, and titled-prefix names (Mr / Mrs / Dr X Y) are all replaced with bracketed placeholders like {{EMAIL}}, {{PHONE}}, {{CREDIT_CARD}} before the response is sent.
• Toggle: clicking the lock disables masking and immediately re-fetches the open conversation + MCP tree so the change is visible.
• Persistence: your choice is remembered for the browser session — if you disable masking and refresh the page, it stays disabled. Closing the tab resets to ON next time you open it.

Tip

Server-side masking means raw PII never reaches your browser when the lock is on. It can’t be inspected via devtools or the network tab. The unmasked path is gated by the same super-admin check as the rest of the page.

The masking is applied to both the conversation panel and every input/output in the MCP call tree, so even when you’re inspecting the raw payload of a tool call you’ll see masked values until you opt-in to the unmasked view.

Coverage notes

The masker is layered: a fast regex pass (microseconds) for structural identifiers (emails, phones, IDs, currency, dates, etc.), plus a POS-tag based name detector (scrubadub’s TextBlobNameDetector, ~5–50 ms per call) for bare names that don’t follow a regex shape. Together they catch:

• Title-prefixed names (“Mr John Smith”) via the regex TitledNameDetector.
• Two-word capitalized names (“Brian Naidoo”, “John Smith”, “Mary Johnson”) via two parallel detectors — a heuristic regex and the POS tagger — both gated by the same finance/geography stop list so common phrases like “Credit Score”, “New York”, “Visa Card”, “January Statement”, “Assistant:” don’t get masked.
• Single capitalised first names mid-sentence (“Brian applied for a loan.”) via the POS tagger’s proper-noun detection.

The trade-off:

• Lowercase bare names (“john smith called yesterday”) still pass through — lowercase prose overlaps too much with normal English to mask safely.
• POS tagging adds ~50 ms per masking call (one-off NLTK corpora load on cold start, hidden by the lazy singleton). Acceptable for the audit page; would be too heavy for the inference critical path if it were ever run there.
• POS tagging can’t separate “Brian” the name from “Brian” the brand — false positives are bounded by the stop list. New non-name capitalised tokens that show up in production should be added to _BARE_NAME_STOP_WORDS in workers/modules/pii_masker.py (single source of truth across both the regex and POS-tag detectors).

If you need stronger coverage (lowercase prose, multilingual, domain-specific entities), that’s an opt-in NER upgrade via spaCy or a transformer model — significantly larger image footprint, currently out of scope.

Customer-id-shaped tokens (e.g. ABC500, cust-123, LN_4567) are matched case-insensitively with a 3-digit minimum. The minimum is what suppresses false positives on natural-language word+number combos like win10 or covid19 — short suffixes pass through, but anything that looks like a real customer / account / loan code gets masked regardless of letter casing.

What’s in the MCP call tree

Each accordion row in the rightmost panel is one inference workflow run. The header shows:

• Start time of the run
• Duration end-to-end
• Status — completed, failed, timed_out, canceled
• Activity count — how many backend steps were involved

Expanding a row reveals the per-activity tree. For each activity:

• Name — the MCP tool or internal step (e.g. mcp_route_prompt, mcp_categorize_data, mcp_summarize_data, mcp_curate_answer, infer)
• Duration
• Status badge — green for OK, rose for failed, amber for timed-out, slate for canceled
• Input — the JSON the activity was called with (collapsible, masked when the lock is on)
• Output — the JSON the activity returned (collapsible, masked when the lock is on)
• Timestamps — scheduled / started / ended
• Error message — when an activity failed, with the failure reason inline

This view answers questions like “why did the agent give that answer?” — you can trace the prompt routing, the categorisation it picked, the summary stage and the final curation, and see what each step received and produced.

Tree vs Graph view

The MCP panel has a Tree | Graph toggle in its header.

• Tree — the per-run accordion view described above. Best for reading individual activity inputs/outputs.
• Graph — a visual reasoning graph rendered with React Flow. Each step is a node, connected top-to-bottom in the order they ran, colour-coded by status, with a side-drawer that opens to show the activity’s full input/output when clicked. Best for seeing the shape of an inference run at a glance — how many iterations of the router/invoke loop fired, where time was spent, where the path branched.

Switching views doesn’t re-fetch the data — both are different visualisations of the same audit document.

LLM exposure visibility

The audit captures the active LLM exposure level for every inference workflow:

• The audit_finalize payload includes an exposure field on every run (full / limited / advisory) — you can see this in the run-level metadata.
• The mcp_curate_answer activity logs the active exposure in its start/end records.
• If the agent’s deterministic leak scanner ever caught the language model trying to quote a row value the project’s exposure level forbids, the run carries a leak_scrubbed marker on the curate activity, and a curate_leak_scrubbed event lands on the Activity page.

Together these let you answer questions like “was this advisory project ever asked for raw rows, and did the agent comply?” without leaving the audit page.

Filtering sessions

The Sessions panel has two controls above the list:

• Search — substring match on session title (case-insensitive)
• Date range — defaults to the last 30 days. Change either bound to widen or narrow.

The “to” date is inclusive to end of day, so picking today as the upper bound includes sessions that ran later in the day.

Where the data comes from

• Users and Sessions are read from the SQL database (dbo.app_user, dbo.customer_super_admin, dbo.ChatSession) scoped to your customer.
• Conversation messages come from dbo.ChatMessage, ordered by their original arrival.
• MCP call tree documents are read from Cosmos DB. They’re written at the end of every inference workflow by the audit_finalize step, which captures the full Temporal event history for that run.
• PII masking is applied on the API server using scrubadub, plus a layer of custom regex detectors (country-specific national IDs, customer-id tokens, currency amounts, percentages, dates, titled / bare names) registered on top of the library’s built-in entity set, plus scrubadub’s bundled TextBlobNameDetector (POS-tag based proper-noun detection via NLTK) for bare names the regex layer can’t pattern-match.

Where to go next

• Managing users — adding users and resetting their passwords.
• Roles & permissions — the boolean permission model that gates everything else in the platform.