
One cell per scenario: result, severity and the state change it saw.
A run selects 1 to 5 events from one provider and sends at most 50 deliveries. With 3 events and every scenario that applies to Stripe, a run sends 17 deliveries.
Rules for every valid delivery
Once, Twice, Shuffled, Retry after error, Burst and Slow budget send valid deliveries: the exact body, signed so the provider’s own verifier accepts it. On any of them:
Before each delivery, a refused connection is retried every 500 ms for up to 5 s, so a dev server restarting after a file change isn’t reported as unreachable. Three unreachable deliveries in a row stop the run as Aborted (
target_lost).
Once
Sends every selected event once, in capture order, 150 ms apart. If the first delivery gets a 3xx or 4xx, the run stops at once as Couldn’t start (baseline_rejected): your handler rejected a correctly signed delivery, which usually means a wrong secret or path. If it is unreachable, the run stops as Couldn’t start (target_unreachable). A 3xx or 4xx on a later Once delivery is non_2xx_on_valid, and the run goes on.
Twice
Sends the primary event, then the same event again 500 ms after the first completes, with the same event id and the provider’s duplicate headers.- With a probe: the state after the second delivery must equal the state after the first. Otherwise:
duplicate_changed_state(High). - Without a probe, two 2xx are Inconclusive (
needs_probe).
Shuffled
Sends the selected events in reverse capture order, which puts every dependent event before the event it depends on.- Needs a probe and a reset: the state after the last delivery must equal the state after Once. Otherwise:
order_changed_state(High).
Late
Sends the primary event with a signature valid for a timestamp tolerance + 60 s in the past: 360 s for Stripe and Standard Webhooks. The body is unchanged. Your handler should reject it.Retry after error
Sends the primary event twice: the second attempt 2 s after the first completes, with the provider’s retry headers. The second attempt is always sent, because a provider retries when it saw a failure or a timeout, and your handler can’t tell a retry from a duplicate.Tampered
Changes one field of the body in the raw bytes, then signs for the original body at the current time, so only the body check can fail. The field is the first that applies:- the first path in the pack’s tamper paths for this event type (numbers become
0, or1if already0); - else the first numeric field whose name contains
amount,total,priceorquantity; - else
"_webhooktrap_tampered":trueis added at the end of the top-level object.
id, type, object, created or ending in _id are never changed. Your handler should reject it.
Unsigned
Sends the primary event with every signature header of the pack removed. The body is unchanged. Your handler should reject it.How Late, Tampered and Unsigned are judged
The finding is
accepts_stale (High) for Late, accepts_tampered (Critical) for Tampered and accepts_unsigned (Critical) for Unsigned. These deliveries don’t count against the budget: providers never send them.
Burst
Sends every selected event at the same moment, from connections opened beforehand. With one selected event it sends two copies of it, with the same event id.- The probe is read once before and once after the whole burst. With a reset: the final state must equal the state after Once. Otherwise:
concurrent_changed_state(High). - If the requests left more than 20 ms apart, the scenario is Inconclusive (
not_simultaneous).
Slow budget
Sends one more valid delivery of the primary event, after the reset, to measure the handler warm.over_budget findings from every scenario merge here.
A latency is over budget when it is strictly greater than the budget. If only the first delivery of the run is over budget, it is reported as the Info finding Slow first delivery (cold_start) instead: development servers often compile a route on the first request.
Probe, reset and Inconclusive
Without a probe, Level 2 judgements can’t run, and scenarios where every delivery got a 2xx are Inconclusive withneeds_probe. Without a reset, the judgements that compare against the state after Once (Shuffled, Burst, the third row of Retry after error) are Inconclusive with needs_reset. See Configuration.