Work / Switchyard
A webhook fails and both sides insist they worked.
Switchyard is the system I built for that disagreement. It keeps the original request, routing decision, delivery attempts, and recovery history together so I can answer a simple question later: what actually happened?
The problem
“We sent it.”
“We never got it.”
Those statements can both be sincere and still leave the incident unsolved. They were observed from different sides of the system.
I wanted a boundary I could inspect directly: did the request reach ingest, what exact bytes arrived, which route matched, whether delivery was attempted, what the receiver returned, and what happened on retry.
What Switchyard keeps
Keep the event and every delivery attempt together.
- Receive
Accept the inbound request at a stable endpoint.
- Commit
Persist the exact event before returning acceptance.
- Inspect
Keep headers, body, timing, and history available later.
- Route
Record which rule matched and why.
- Deliver
Send through a controlled delivery path.
- Recover
Retry or replay without erasing the failed attempt.
- Export
Create a sanitized receipt that can travel with the escalation.
A failure I exercised
HTTP 500, diagnosis, repair, replay.
I built a disposable receiver that could fail on demand, then used only the documented operator path to work the incident.
- 01Event accepted
The original webhook is committed and remains inspectable.
- 02Receiver returns 500
The failed attempt stays attached to the event.
- 03Diagnose
Operator commands identify the failed delivery and export an incident record.
- 04Repair + retry
The receiver is fixed and the same preserved event is replayed successfully.
The recorded walkthrough completed in 112 seconds against a 180-second bound. Its sanitization check passed and the hard invariant audit remained at zero.
One design choice that matters
A restart should not erase the incident.
PostgreSQL is the durable source of truth. Oban handles durable jobs. PubSub tells interfaces that durable state changed; it does not become the state itself.
That is more machinery than an in-memory relay, but it matches the problem I care about. If the process disappears during an incident, the evidence and recovery path still need to be there when it comes back.
What I actually measured
The soak test was useful because it was not perfect.
recorded soak duration
events accepted
sustained on the recorded M1 Pro environment
Of 142,120 accepted events, 142,110 reached succeeded. Hard invariants stayed at zero and no duplicates were recorded, but the strict harness still classified the run Invalid because accepted and completed counts differed and its error count was non-zero.
I kept that result because it is more informative than rounding the story up to “the benchmark passed.” It says the system stayed healthy under the tested load while also preserving the fact that not every offered event completed successfully.
LIMIT The throughput belongs to the named hardware, topology, software, configuration, and duration. It is not a capacity promise.
Why this project fits me
It is a support problem with software wrapped around it.
I have spent a lot of my career between a customer report and the system that can explain it. Switchyard is the same kind of work at a webhook boundary: preserve the observation, narrow the failure, recover deliberately, and leave the next person with a history they can inspect.
The project is public, self-hosted, and intentionally explicit about what its evidence does and does not prove.