Write-up
Switchyard: voice agent operations console
An operations console for AI phone agents: per-turn latency, a question log from day one, and a CRM write-back that is only done once a read-back confirms it.
- Company
- Personal Project
- Role
- Sole architect and developer
- Period
- Aug 2026
- Duration
- 4 days
- Team
- Solo
- Voice AI
- Observability
- Hexagonal Architecture
A walkthrough of the running application: the overview with an annotated speech-to-text degradation, a live call showing where each turn's milliseconds went, and a CRM write-back that fails with a 503, retries under the same idempotency key, succeeds, and is then verified by a read-back.
Stack
- TypeScript
- Next.js
- Hono
- Drizzle ORM
- PGlite
- Server-Sent Events
- Vitest
- Playwright
The Challenge
Success is reported, not confirmed
Wiring up an AI phone agent is the easy part. What is missing is the operations surface: whether it answered, what it told people, when a human took over, and whether the appointment actually landed in the business's system. Some evaluation tools do check backend state once, after a simulated call. What is absent is the same check on production calls, with a retry under a stable key and a reconciliation sweep behind it.
The transport cannot express failure
Vapi's tool documentation: "Always return HTTP 200, even for errors. Any other status code is ignored completely." (docs.vapi.ai/tools/custom-tools-troubleshooting)
Latency is one number per call
No per-turn breakdown
What the agent could not answer is never recorded
No question log, no miss log
The model is trusted with authority
No allow-list between a model request and an action
Solution Design
The model controls language, application code controls authority: Hexagonal, with the composition root as the only file that names a concrete adapter. Call events arrive through one intake port as a discriminated union; the core reduces them into calls, turns, latency spans and write-backs, and derives disposition and containment itself so no platform adapter can set them. Adding a platform is one adapter folder, and removing one is deleting a folder.
A write-back is not done until a read-back says so
- Why
- A 200 is the vendor's claim about its own state. The record either exists afterwards or it does not, and only a read answers that.
- Trade-off
- An extra call per write, and two terminal states to explain instead of one word for success.
- Evidence
- Every success is followed by a read-back that stamps a verified status. A write that had failed and was later landed by the reconciliation sweep under the same key ends as recovered, so 'we checked' and 'we repaired' never collapse into 'fine'.
The idempotency key is derived, not generated
- Why
- A retry has to be provably the same request. A key hashed over the call, the action and the canonicalised arguments is stable across attempts and across a process restart.
- Trade-off
- Argument canonicalisation becomes load bearing: a change in how arguments are ordered or serialised changes every key.
- Evidence
- Tests pin key stability for identical arguments and divergence for different ones, and assert that the same key sent twice creates exactly one record.
The retry schedule is asserted against a virtual clock
- Why
- A backoff that is roughly right is untestable, and a demo whose timings move between takes cannot be filmed.
- Trade-off
- Every component that reads time takes a clock port. There is no ambient current time anywhere in the system.
- Evidence
- Attempts are asserted at exact offsets with the attempt log persisted per attempt, and exhausted retries raise an alert rather than failing quietly.
The model requests, the application decides
- Why
- The interesting failure in voice agents is not a wrong sentence, it is a wrong action. Keeping the allow-list in application code makes the boundary inspectable and stops it moving whenever the prompt is edited.
- Trade-off
- The product cannot grow abilities by prompt alone. Every capability is an explicit entry carrying its own retry and reconciliation policy.
- Evidence
- The settings screen renders that allow-list with each action's policy beside it.
One labelling regime for latency
- Why
- The seeded figures are exact in the data, but a mix of exact and approximate labels on screen invites a question about why this one is exact, at the worst possible moment.
- Trade-off
- Approximation is shown even where the number is in fact exact.
- Evidence
- Every latency figure in the console carries the same approximation prefix, including the seed adapter's.
Old system vs. new architecture
| Layer | Old | New | Reason |
|---|---|---|---|
| Write-back | Trust the 200 | Read-back verification | Existence is the only proof |
| Retry identity | A new request per attempt | Derived idempotency key | A retry must be the same request |
| Latency | One duration per call | Per-turn spans by stage | Shows where the time actually went |
| Demo data | Fixtures per screen | One seeded workspace | Screens agree with each other |
| Time and randomness | Ambient | Ports: virtual clock, seeded generator | Renders repeat exactly |
Execution Strategy
Core, seed adapter and console
Days 1 to 2Boundary guard first
A lint config and a test that greps the tree landed with the scaffold
Outcome
The composition root stayed the only file naming an adapter
Deterministic substrate
A seeded generator and a virtual clock behind ports before any domain code
Outcome
Every later feature was reproducible by construction
Write-back pipeline
Derived idempotency keys, a pinned retry schedule, read-back verification, and a mock CRM with fault injection
Outcome
The failure path became a demonstrable feature rather than an untested branch
Logs from day one
Question clustering over normalised text, plus rule-based categorisation of misses
Outcome
Recurring questions surface as clusters, not rows
The filming and accessibility gate
Days 3 to 4Run the suite twice
The browser suite runs once normally and once with reduced motion emulated
Outcome
A reader who turns motion off loses nothing
Computed accessibility, not asserted
Contrast, focus rings, keyboard reachability, target size and layout shift are measured in the browser per route
Outcome
Findings arrive from the page, not from a checklist
Console cleanliness
Every screen fails its test on any browser console error or warning
Outcome
Nothing surprising appears mid-recording
Impact & Results
Business outcomes
- The failure path is the demo
- The scripted call fails a write with a 503, retries under the same idempotency key, succeeds, and is then verified by a read-back. Trust comes from watching a failure handled, not from a screen with no failures on it.
- One command, no keys, no network
- The database is Postgres compiled to WASM inside the process, the fonts are self-hosted, and the demo workspace sits behind the same intake port a live platform would use.
- Adding a platform is one folder
- Intake is a single port over a discriminated event union, and the composition root is the only file naming a concrete adapter. Both rules are enforced by lint and by a test that greps the tree, so they cannot rot quietly.
- No live platform adapter yet
- Everything demonstrated runs against the seeded workspace, which is labelled as sample data on every screen.
Key Lessons
A green suite is not a green product
Four defects reached the recording through a full unit, contract and browser suite, all of them in the seams the tests had quietly stubbed out: a clock each test injected, a stream each test opened once, adapter state each test reset. Tests that isolate a component perfectly are the ones most likely to miss what only happens when two of them overlap.
- Testing
- Realtime Systems
The wire state cannot tell you what the server meant
A browser reconnects after a deliberate close and after a dropped connection, so the connection state reads identically for both. Anything the client needs to know about intent has to be carried in the protocol. This one was found by reading recorded frames, which is a slow and unrepeatable way to find a bug, and the only one that worked.
- Realtime Systems
- Debugging
A 200 is a claim, a read-back is proof
The most valuable screen in this product is the one showing a write that failed, retried under the same key, and was then confirmed to exist. Splitting the terminal states into verified and recovered costs an extra concept to explain and is worth it, because collapsing them hides which writes needed repair.
- Integrations
- Reliability
Determinism has to be a port, not a discipline
Time, randomness and identifiers all arrive through ports, which is more ceremony than reaching for the ambient version. It buys a demo that renders identically between takes and a retry schedule that can be asserted at exact offsets. The cost is that one forgotten ambient call anywhere reintroduces the flake, which is why the boundary is enforced by a test rather than by care.
- Testing
- Architecture
Authority belongs in code, not in the prompt
Separating what the model may say from what the application will do gives a boundary you can show a non-technical buyer, and it does not move when the prompt is edited. The cost is real: every new capability is an explicit entry with its own retry and reconciliation policy, so the product cannot grow abilities by prompt alone.
- Voice AI
- Agent Design