Write-up
First Light: overnight automation triage
A connection-health monitor for automation workflows and cron jobs. It turns the night's failures into a morning brief, and replays held work without doing it twice.
- Company
- Personal Project
- Role
- Sole architect and developer
- Period
- Aug 2026
- Duration
- 4 days
- Team
- Solo
- Observability
- Automation
- Hexagonal Architecture
A walkthrough of the running application: the overnight digest, the four connection states including the two most products fake, a replay that suppresses an item already processed, and the run it links to as proof.
Stack
- TypeScript
- Next.js
- Hono
- Drizzle ORM
- PGlite
- Server-Sent Events
- Vitest
- Playwright
The Challenge
Failure without an audience
Automations do not fail loudly. A token expires at 02:14 on a Tuesday, runs keep failing until somebody notices, and the notice usually arrives from the customer rather than from the system. The monitoring most small teams have is an inbox.
Failure is discovered by the customer
No signal until someone complains
Dashboards fill gaps with zeros
An unknown metric renders as 0%
Re-running held work risks doing it twice
A retry carries no memory of the item
One broken integration hides the rest
A single adapter throw aborts the refresh
Solution Design
Decisions in a core that cannot reach the network: Hexagonal. A pure domain package owns every decision: whether a connection is healthy, whether a held item is a duplicate, what the operator should do next. Adapters own storage, intake, transport and rendering. Time arrives through a clock port, so any decision can be tested at any instant with no wall clock and no database.
A pure core, enforced twice
- Why
- The health rules have to give the same answer whether runs arrive from the seeded narrative, a platform poller or a webhook script, and have to be testable without a database, a network or a clock.
- Trade-off
- Indirection. Adding a capability means touching a port, an adapter and a service, which is slower than a route handler reading the database directly.
- Evidence
- A test reads the core's own sources and fails on Node built-ins, adapter imports, fetch, environment access and bare date construction. A lint rule fails the same set while you type.
An unknown metric is null, never zero
- Why
- Zero is a measurement. Rendering it when the truth is that not enough runs have been seen turns an absence into a claim, and operators act on claims.
- Trade-off
- Four states instead of one percentage, so every screen needs a design for 'we do not know' rather than a number and a sparkline.
- Evidence
- Health derivation returns healthy, degraded, not connected, or insufficient data, each with a written reason the interface renders verbatim. Every branch has its own test.
The replay guard is item-level, not run-level
- Why
- Run-level idempotency cannot protect a replay: the held item's original run and the run that already processed it carry different keys by construction, so the only stable identity is the item.
- Trade-off
- Intake has to record which items each successful run processed. That is extra write volume and one more thing every future adapter must supply.
- Evidence
- Before executing anything the replay service looks up the automation and item key pair. A hit resolves the item as suppressed and links the run that already did the work; a test asserts the replay port is never called on that path.
Postgres in WASM for the demo, hosted Postgres when deployed
- Why
- A demo backed by a mock store exercises code the deployment never runs. Compiling Postgres to WASM means the demo and the store tests run the real queries: unique constraints, JSON columns, conflict targets.
- Trade-off
- A WASM database sits in the test path, so the store suite takes seconds where an in-memory fake would take milliseconds, and version parity with a hosted instance is close rather than guaranteed.
- Evidence
- One command boots the API with the database inside its own process: no containers, no credentials, no network.
Status is never colour alone
- Why
- The screen is read at 08:00 by someone deciding what to touch first, sometimes on a projector, sometimes in a screenshot pasted into a chat.
- Trade-off
- Every state costs a drawn glyph, a distinct shape and a visible label, so an indicator takes more room than a coloured dot.
- Evidence
- The status component offers no label-less mode, and the states stay distinguishable in greyscale.
Old system vs. new architecture
| Layer | Old | New | Reason |
|---|---|---|---|
| Demo data | Mock store | Postgres in-process | The demo runs the deployment's queries |
| Unknown metric | 0% | null, shown as insufficient data | An absence is not a measurement |
| Status | Colour | Glyph, shape and label | Readable in greyscale |
| Live updates | Polling | Server-sent events with resume | No broker, and no silent gap on reconnect |
| Idempotency | Run key | Automation and item key | A replay is a different run doing the same work |
Execution Strategy
Core, store and the console
Days 1 to 2Boundary first
The purity test and the lint rule landed before the domain code they protect
Outcome
No input or output ever leaked into the core
Four honest states
Health derivation with explicit thresholds and a written reason per branch
Outcome
Every screen has a design for 'we do not know'
Two idempotency rules
Run-level on intake, item-level on replay, named and tested separately
Outcome
A replay that refuses to double-process
Deterministic narrative
A seeded 30-day dataset written at boot, with a frozen clock for filming
Outcome
Identical renders between takes
Intake beyond the seed
Day 3Contract first
Execution mapping frozen against captured fixtures before the poller existed
Outcome
The mapping is testable with no external instance running
Catch-up and isolation
A per-source high-water mark persisted across restarts, each source settling independently
Outcome
One broken integration cannot abort the cycle
Reporting in, and the safety net
Day 4Signed webhook intake
A rate-limited endpoint behind a schema contract, with keys hashed at rest and shown once
Outcome
Any script can report in without a plugin
Channel-agnostic alerts
The core emits a severity, a sentence, a next step and a deep link; formatting lives in the channel adapter
Outcome
Adding a channel does not touch the policy
Hourly sweep
Expands expected schedule slots, re-runs what is missing, and records anything undecidable as a question rather than resolving it
Outcome
A safety net that never guesses
Impact & Results
Business outcomes
- One command, no credentials, no network
- The database is Postgres compiled to WASM and lives inside the API process, so the demo boots with an empty environment.
- The duplicate guard can be demonstrated rather than asserted
- The refusal is a database lookup on the automation and item key, and the row it produces links to the run that already did the work.
- The honesty rule is a domain decision, not a presentation one
- The reason sentence an operator reads is produced by the core and rendered verbatim, so it cannot drift between the API and the screen.
- Not deployed, and no operators
- Everything here runs locally. Nothing on this page is a claim about production behaviour.
Key Lessons
Idempotency is not one rule
Run-level and item-level idempotency protect different things, and the second cannot be derived from the first: the run that already did the work and the run that failed carry different keys by construction. Naming them separately was the change that made the replay guard correct.
- Distributed Systems
- Automation
A missing number is not a zero
The rule sounds obvious and costs real work: four states instead of one percentage, a written reason for each branch, and a design for every state on every screen. It is also the part an operator has to trust before any of the rest matters.
- Observability
- Product Engineering
A boundary nothing enforces is a preference
The core's purity is checked by a test that reads its own sources and by a lint rule that fails the same thing while you type. The cost is indirection: every capability touches a port, an adapter and a service. That price buys one implementation of each rule across the demo, the tests and the deployment.
- Architecture
- Testing
Real Postgres in the test path is worth the seconds
An in-memory fake would run the store suite in milliseconds and would never exercise a unique constraint or a wrong conflict target. Running the same schema in WASM costs a slower suite and leaves a residual version-parity risk, which is why a smoke test against a hosted instance is scheduled work rather than an assumption.
- Testing
- Databases
The evidence here is design, not usage
Everything defensible about this project is a property its tests enforce or something visible on camera. There are no users, no production incidents and no operator feedback, and the demo dataset comes from a fixed seed, so its figures describe a fixture and measure nothing.
- Honest Claims