Environment incidents
How kinora recognizes a run that failed because the app under test was down, and keeps it out of histories, baselines and alerts.
When the environment under test is down, every test fails at once: staging refuses connections, a DNS name stops resolving, the browser crashes on a starved runner. None of those failures says anything about the tests or the code. kinora flags such a run as an environment incident and keeps it out of everything that compares runs.
How a run is recognized
When a run finishes (for a sharded run, when its last shard lands), kinora reads its failures. The run is flagged when at least 3 tests failed, at least 80% of the executed tests failed, and either:
| Rule | 70% of the failures or more were |
|---|---|
| Host unreachable | Navigating or connecting to the app’s host, and getting no answer |
| Browser crash | The browser crashing, closing, or failing to launch |
A failure counts as reaching the app’s host when it is a network error (net::ERR_CONNECTION_REFUSED,
a name that does not resolve, a reset connection, ECONNREFUSED from a request fixture) or a
page.goto that timed out, and the address it names is the host of a Playwright project’s
baseURL. The reporter sends the baseURLs, and a
CLI upload takes them with --base-url. Without one (Vitest, a CLI upload without
the option), the host most failures named stands in for it.
What never counts: a third-party host going down (a CDN, an analytics script), and a failing assertion, however many tests it breaks. A login page broken by a commit is a regression, not an incident. A test that timed out and then saw its browser close is a timeout, not a crash.
What a flagged run changes
- The run page says Environment incident · not counted, with the reason (41 of 44 tests failed, 39 of them navigating or connecting to staging.example.test (connection refused).) and whether a rule or a person decided. Run lists show an Incident badge.
- Test histories, flaky and broken flags, and failure causes leave the run out. A test that failed only in the incident does not become flaky or broken.
- Baselines skip it: the next run is compared with the last run before the incident, so it does not report every test as fixed.
- Alerts send one message naming the outage instead of the newly failing
tests, on every policy, including on regression. Webhooks carry it under
incident, with emptynewlyFailingandnewlyFlaky. - The PR comment says the run was an environment incident and lists no regressions. The reporter and the CLI also print a warning with the reason.
- MCP:
list_failuresandget_runinclude the incident, so an agent does not read an outage as a regression.
The run itself stays: its tests, errors and traces are kept like any other run’s, and the analytics outcomes still count it. The analytics flaky and failure rates leave it out, so an outage does not hide a slow drift.
Stopping a run before it starts
With @kinora/reporter/check-base-url in
globalSetup, Playwright checks that each baseURL answers before any worker starts, and stops
the run when one does not. No test runs, so the outage costs no CI time and no run reaches the
dashboard.
Marking and clearing by hand
The rules can miss an outage or flag a real regression. On the run page:
- Mark as incident on a run with failures the rules did not flag (the payment sandbox was down, a shared account was locked). It is left out the same way.
- Clear the flag on a flagged run whose failures are real. It counts again everywhere.
The decision is kept on the run, and a shard landing later never overrides it. The run page says who decided.
Limits
- An outage that hits only part of the suite (one service down, 30% of the tests failing) stays below the thresholds: mark it by hand.
- Marking or clearing a run does not send the alerts it skipped, or withdraw the ones it sent.