Evidence-backed CI governance

Make red mean broken again.

FlakeGate finds tests that pass and fail on the same code, keeps governed flakes out of the blocking path, and makes every unknown regression stay red.

Start locally View repository
  • One binary
  • Zero servers
  • No skipped tests
Test evidence entering a gate and separating into trusted, quarantined, and blocking paths
0 FP · 0 FN labeled replay
17 / 17 safety controls
26,765 labeled tests
869,550 raw executions
10,687 IDoFT records
6 release targets

The trust problem

A red build should carry information.

Once “just rerun CI” becomes routine, developers and coding agents can no longer tell a random test failure from a real regression. FlakeGate restores the distinction.

Without governance

Rerun until green

  • Random failures consume engineering time.
  • Blanket quarantine turns into permanent debt.
  • Real defects can be rerun past.
  • Agents act without reliable history.

With FlakeGate

Decide from evidence

  • Same-code contradictions prove flaky behavior.
  • Every exception has an owner and expiry.
  • Unknown and expired failures remain blocking.
  • Humans and agents share one verdict.

One closed loop

Detect. Govern. Gate. Repair.

Tests keep running throughout the lifecycle. FlakeGate changes the CI decision, never the user’s test code.

01

Ingest

Read standard JUnit XML from every CI run into one cacheable SQLite file.

02

Detect

Rank intra-run contradictions, same-commit flips, and recurring oscillation.

03

Govern

Commit a reason, owner, start date, and expiry for every temporary exception.

04

Gate

Unblock active known flakes while unknown regressions and expired entries stay red.

Two identical code fingerprints producing opposite pass and fail outcomes

Evidence, not guesswork

Same code. Opposite outcomes.

A pass and failure from the same run or fixed revision is definitive evidence. A one-off failure is not. FlakeGate keeps that distinction explicit.

  • 1Intra-run pass/fail contradiction
  • 2Same-commit flip across runs
  • 3Weighted recurring oscillation
A temporary quarantine chamber with owner, expiry, repaired return, and re-blocking paths

Quarantine with an exit

A repair queue, not a graveyard.

Active known flakes stay visible without blocking delivery. Repaired tests return to the trusted path. Unresolved entries automatically block again after expiry.

Required owner Required reason Start date Expiry date

Evidence, not just claims

Benchmarked on published research artifacts.

Each number is tied to a reproducible task and an explicit limitation. Positive-only coverage is never presented as precision.

Pinned research archives flowing through verification into reproducible results
Pinned artifacts Executable replay Documented limits

FlakeFlagger · ICSE 2021

828 TP · 0 FP · 0 FN

26,765 labeled tests in the decisive detector-semantics replay.

Raw Surefire replay

869,550 executions

47,430 XML reports from 7,905 valid fixed-revision runs.

IDoFT / iDFlakies

10,687 / 10,687

Evidence recall across 916 projects and 14 observed categories.

Operational controls

17 / 17 / 17 / 17

Retry, active quarantine, injected regression, and expiry checks.

Modeled full-run probabilities assume all labeled flakes run and outcomes are independent. IDoFT is positive-only, so its result demonstrates evidence coverage, not precision.

Read the methodology Run the workflow

Built for real CI

Local-first by default. Automatable everywhere.

Universal input

One format: JUnit XML

Work across pytest, Jest, Surefire, Gradle, gotestsum, RSpec, and other producers without per-framework FlakeGate plugins.

PythonJavaGoJavaScriptRuby
Private core

Zero-server operation

History stays in one local SQLite file carried by the CI cache.

Agent-ready

JSON + local MCP

Humans and coding agents consume the same trust decision.

Safety first

False-green guardrails

Empty reports, unknown failures, and expired quarantines block.

Operations

Doctor + preview-first prune

Diagnose locally and inspect retention impact before deleting history.

Quick start

Put the gate after your test runner.

Keep the test command observable, let it emit JUnit, and make FlakeGate own the final CI exit code.

Terminal 
go install github.com/xiao-zi-chen/FlakeGate/cmd/flakegate@latest

pytest --junitxml=report.xml || true
flakegate gate \
  --config .flakegate.yml \
  --summary flakegate-summary.md \
  report.xml
.flakegate.yml 
version: 1
quarantine:
  - test: tests.core::test_timing
    reason: "timing: hard-coded sleep"
    owner: test-platform
    since: 2026-07-15
    expires: 2026-08-15
Try the entire loop without a network.

PowerShell and POSIX demos are included in the repository.

Open the demo

Your CI can be trustworthy again

Stop rerunning past the signal.

Start with one JUnit report. Keep every exception visible, owned, and temporary.

Explore FlakeGate Read the repair playbook