Xping for .NET

Your coding agent can't tell a flaky test from a broken one.

Xping records every dotnet test run — local ones included — so "is this test flaky?" gets answered from history instead of from one red run.

Recording and local reports are free, need no account, and make no network calls.

Why this exists

dotnet test is amnesiac. Every run starts from zero and discards everything the last one knew. A test that failed on Tuesday and passed on Wednesday leaves no trace anyone can point at on Thursday, so "is that one flaky?" gets answered from memory and settled by re-running until it goes green.

That was already expensive when a human was doing it. Now an agent is. Hand an agent a failing test and a single run to learn from, and it will make the test pass. With one data point, a real regression and a flake look identical — so it fixes what it can see:

It does this	And you get
Loosens the assertion	A test that still runs, but stopped checking the thing that broke
Adds a retry	A real, reproducible failure that now passes on attempt three
Bumps the timeout	The same race condition, just harder to hit in CI
Moves the test into its own `[Collection]`	A green suite that now runs those tests serially — the shared-state bug is untouched, and every build is slower
Mocks the dependency that failed	A test that passes deterministically because it no longer touches the code that broke

Every one of those is a reasonable move on one data point. Nobody notices until the suite stops meaning anything.

Xping is the accumulation layer underneath. Nothing about your test code changes.

What is Xping?

Xping SDK — a lightweight library that hooks your test framework (NUnit, xUnit, MSTest) and records each execution: outcome, duration, environment. Under 5 ms per test. Targets netstandard2.0, so it runs on any supported .NET version.

Xping CLI — the xping global tool. Reads the recorded history back and tells you which tests behaved consistently and which didn't — to you in the terminal, and to your agent as structured JSON. Targets net10.0.

Xping Cloud — the hosted platform at app.xping.io. Pools history across every machine, branch, and CI job, then scores it: confidence per test, evidence sufficiency, root-cause categorisation, trends, and PR comments on GitHub. Currently invite-only.

The SDK and CLI are two halves of one thing — the SDK alone writes history nothing reads. Cloud is optional and additive.

Local vs. Cloud

	Local	Cloud
Setup	SDK package + CLI tool	+ an upload key
Account	Not required	Required (invite-only)
Your data	Stays in `.xping/` in your repo	Uploaded to Xping Cloud
History	Your machine, your runs	Every machine, every branch, CI included
You get	Observations and evidence from local runs	Confidence scores, evidence sufficiency, trends, PR comments

The local CLI reports what it observed. It does not assign confidence scores or name causes — a handful of runs on one machine isn't an evidence base strong enough to support that kind of claim, and we'd rather show you the runs than a number that implies more certainty than the data has. Scoring lives in Xping Cloud, where there's enough history across machines, branches, and CI to justify it.

	Local	Cloud
Run-by-run pass/fail history	✓	✓
Inconsistent and newly-failing tests	✓	✓
Consistently-failing tests (real bugs)	✓	✓
Duration and environment capture	✓	✓
JSON envelope for agents and scripts	✓	✓
Per-test confidence score	—	✓
Evidence sufficiency (ESS)	—	✓
Root-cause categorisation	—	✓
Reliability trends over time	—	✓
Cross-environment comparison	—	✓
History across CI, branches, teammates	—	✓
GitHub PR comments	—	✓

Start local. Nothing leaves your machine, and you can add an upload key later without changing a line of test code.

Quick Start

1. Install both pieces

dotnet add package Xping.Sdk.XUnit    # xUnit v2; or Xping.Sdk.XUnit.V3 / Xping.Sdk.NUnit / Xping.Sdk.MSTest

dotnet tool install -g Xping.Cli      # puts `xping` on your PATH

The CLI needs the .NET 10 SDK or runtime. The SDK packages target netstandard2.0, so your test projects can stay on any supported .NET version.

2. Turn on tracking

Add one line to AssemblyInfo.cs. Which one depends on the xUnit package your project references:

[assembly: TestFramework("Xping.Sdk.XUnit.XpingTestFramework", "Xping.Sdk.XUnit")] // xunit (v2): Xping.Sdk.XUnit
[assembly: TestFramework(typeof(Xping.Sdk.XUnit.V3.XpingTestFramework))]           // xunit.v3: Xping.Sdk.XUnit.V3

xUnit Quick Start →

Add one assembly-level attribute:

[assembly: XpingTrack]

NUnit Quick Start →

Inherit from the base class:

[TestClass]
public class MyTests : XpingTestBase { }

MSTest Quick Start →

3. Run your tests — more than once

dotnet test

One run tells you nothing about reliability. A dozen tells you plenty. Run them the way you normally would — including the local runs that never reach CI, which is where most of the evidence has always been thrown away — and let the history build up. Cross-run flakiness detection needs at least three runs.

4. Ask what happened

xping report

Xping · SampleApp.MSTest · 9 runs · 2026-08-20 07:52 → 09:02 · main@bdbafba
17 tests · 14 healthy · 3 flagged

```
NEEDS ATTENTION (4 high)                               most severe first
────────────────────────────────────────────────────────────────────────

1.  HIGH  flaky
    SampleTests.FlakyTest_PassesOnRetry
    failed 9 of 18 executions (50%) in 9 of 9 runs, 1 failure mode
    evidence moderate | -env-cluster | f_2b84a621 | ...SampleTests.cs:96

2.  HIGH  masked by retry                                same test as #1
    SampleTests.FlakyTest_PassesOnRetry
    passed on retry 9 times in 9 of 9 runs, up to attempt 2
    evidence moderate | -env | f_e98db1e6 | .../SampleTests.cs:96

3.  HIGH  always failing
    SampleTests.ThrowingTestIsTracked
    failed 9 of 9 executions (100%), one failure mode:
    System.InvalidOperationException
    evidence low | -env-cluster | f_c1774d82 | .../SampleTests.cs:65

4.  HIGH  flaky
    SampleTests.FlakyTest_RaceCondition_FailsIntermittently
    failed 4 of 9 executions (44.4%) in 4 of 9 runs, 1 failure mode
    evidence low | -env-cluster | f_d24c5aa9 | .../SampleTests.cs:118
```

rates: the marker on each finding says which runs its percentage was
counted out of; compare two only where the markers match.
https://docs.xping.io/cli/command-reference.html#the-population-marker

Detail of row 1: xping report --id f_2b84a621 --assembly SampleApp.MSTest

No API key, no signup, no network calls. Everything lives in .xping/ on your machine.

Running Without an Account →

The CLI surface

xping report                    # findings from recent runs
xping report --format json      # the versioned envelope, for a script or an agent
xping report --summary          # one line, for a chat message or a CI step title
xping report --fail-on high     # exit non-zero when a high finding shows up
xping report --runs 50          # widen the window (or --since <sha|yyyy-MM-dd>)
xping report --kind flaky       # restrict to one or more finding kinds
xping where                     # show where local runs are stored
xping clear                     # delete recorded runs

Full flags, finding kinds, and the JSON schema: CLI Command Reference.

Feeding it to your agent

xping report --format json emits a versioned envelope — findings, evidence levels, run counts, failure signatures — which is what makes the difference between an agent guessing and an agent knowing:

The agent hits a failing test. Instead of rewriting the assertion on one red run, it asks Xping what this test's history looks like.
Xping answers with evidence. Failed 34 of 34 runs, one failure signature, evidence high — or failed 4 of 40, three signatures, only ever in parallel.
The agent acts on it. A failure that consistent isn't flake, it's a bug, and the test was right to catch it. A single local failure against a long clean history is noise: run it again, don't touch the test.

What comes back is evidence, not a verdict. The agent still decides.

Connecting to Xping Cloud

Set one environment variable. Nothing else changes — same packages, same attributes.

export XPING_APIKEY="your-api-key"

Your project is named after your test assembly, so a solution with several test projects gets one Xping project each — created automatically when tests first run. Set XPING_PROJECTID (or ProjectId in appsettings.json) to report them all as a single project instead; see the configuration reference.

Runs upload as they finish, so history pools across every machine, branch, and CI job — a test that only fails on a teammate's laptop still counts.

Xping Cloud is currently invite-only. We're running a small, high-touch pilot while the scoring model settles. Request access — or keep working locally, which stays free and account-free regardless.

XPING_APIKEY is an upload-only key. Export it from the environment on every dev machine and in CI; never commit it. It can write test runs and nothing else, so the key sitting in your CI secrets is not a way into your data.

Reading the team's scored history is a person's action. Sign in once per machine, and xping report adds Cloud confidence to the rows it already prints:

xping login
xping report

See xping login and Cloud data.

For CI setup (GitHub Actions, Azure DevOps, Jenkins, GitLab), see CI/CD Integration.

How It Works

        Your test project  (xUnit · NUnit · MSTest)
                        │
                        ▼
            Xping.Sdk.<framework> adapter
                        │
                        ▼
                  Xping.Sdk.Core
         tracking · environment detection
                        │
      ┌─────────────────┴─────────────────┐
      │  no upload key                    │  XPING_APIKEY set
      ▼                                   ▼
.xping/ (local store) ─ batched upload ─▶ Xping Cloud
      │                                   │  scoring · root cause · trends
      │                                   │
      │                                   ▼
      │                              app.xping.io
      ▼                                   ▼
      └──────────────▶ xping report ◀─────┘
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
        you, in a terminal        your agent, via --format json

Adapters are thin — they hook the framework's execution pipeline and hand results to Xping.Sdk.Core, which owns collection, environment detection, and delivery.

Two things are worth reading off that diagram:

The CLI is the only reader of the local store. .xping/ is an implementation detail; xping report is the interface. See Local Store.
The terminal isn't the only surface. Not everyone who cares about the suite runs it. Developers get the report one command away; QA, leads, and product open app.xping.io and see the same evidence with nothing installed.

Configuration

Configure via environment variables, appsettings.json, or programmatically. Environment variables are recommended for the API key — it should never reach source control.

# Cloud only
export XPING_APIKEY="your-api-key"

# Optional: pin every test assembly to one project instead of one project each
export XPING_PROJECTID="your-project-id"

# Optional, either way
export XPING_ENABLED="true"
export XPING_BATCHSIZE="100"
export XPING_CAPTURESTACKTRACES="true"

Without XPING_APIKEY, the SDK writes to .xping/ and makes no network calls. Full options in the Configuration Reference.

Your Data

Per test execution — name and fully qualified name, outcome, duration, start and end timestamps (UTC), error message and stack trace on failure, the declared timeout budget when the test sets one, categories and traits. A test the framework killed for overrunning its timeout is recorded as a timeout, not folded into ordinary failures.

Per environment — OS and version, .NET runtime version, machine name, CI platform detection, build and branch information from the CI environment, network metrics, and the machine's time zone and UTC offset at the start of the run — which is what lets a failure be placed at a local time of day rather than only at a UTC instant.

What never gets recorded — no source code, no assertion values, no credentials or secrets, no personally identifiable information.

Locally, everything is written to .xping/ in your repository and stays there. Add .xping/ to your .gitignore — the history is machine-local and isn't meant to be shared through version control. To start over, run xping clear. For Cloud, data is transmitted over HTTPS; stack trace capture is configurable and retention is set per workspace.

What Xping Helps You Find

Patterns that show up in accumulated execution history:

Race conditions — intermittent failures with no code change between runs
External dependencies — failures that track network or service availability
Shared state — tests that pass alone and fail in a suite
Time-based flakiness — failures clustered in one local time of day, at weekends, or on one side of a daylight-saving change
Non-deterministic data — random inputs that occasionally hit an edge case

Working locally surfaces the behaviour; Xping Cloud attributes the cause. See Common Flaky Patterns for worked examples.

How Cloud confidence scores work: Xping analyses test execution history across six weighted factors — pass rate (35%), execution stability (20%), retry behaviour (15%), environment consistency (15%), failure patterns (10%), and dependency impact (5%) — producing a 0–1 score. ESS is the effective sample size: how many independent runs stand behind that number. See Understanding Confidence Scores.