How the Assayer tests

Cases#

Code finds a repository's cases from its own files, at the commit tested:

Settings > Test cases you set turns a found case off or on, gives it another command, or adds one of your own: by the Project's name as Castellan's Settings have it, the Case, its Command, and Run it.

A project with no test command is tested by the cases it suggests.

Tests#

Each case's tests are read from its runner:

A runner it can't read still gives the case's pass or fail, and its output.

Each repository, case and test shows its last run, passed or failed, how long it took, its last few runs as dots, and the failing output. A test that passed and failed on the same commit with the same tools is marked flaky.

A test run#

A test run is every case that's on, one after the other.

• In a throwaway worktree under the Assayer's data folder, detached at the commit tested. Your checkout's working tree is never touched.
• At below-normal priority, so the PC stays yours.
• Stopped after 20 minutes for the whole run (Stop a test run after, a setting), with everything it started. A run stopped this way counts as a failure.
• A Node project borrows your checkout's node_modules when the commit's package-lock.json matches the checkout's, through a link that's taken away before the worktree is removed, so your node_modules is never emptied. When they don't match, installing is left to the case's command: npm ci && npm test, say.

What changed#

Each run notes the versions of the tools it used: Git, and Node.js, npm, pnpm, Yarn, Python, Rust, .NET, Go, Java or the Visual Studio build tools, as the repository needs.

When a case goes from passing to failing, the run says what changed since the last passing run: the commits between them, and the tools whose versions moved.

Put a tool back

When it's the same commit and only the tools changed, the toolchain is the likely cause. The page offers each tool's last good version back, through the version manager you already use (nvm-windows, pyenv-win, rustup) or winget. Tick the ones to put back, and press Put back. The Assayer never changes a tool on its own.

Faster tests: only what a change reaches#

A branch of yours is a change against the project's branch, so its unit tests run only the test files that change reaches (Test only what a change reaches, on by default):

• node:test (a test script of the form node --test "<glob>"): the test files that changed, the ones whose imports reach a changed file (however deep), and the ones that name a changed file. A change that only raises version lines, or Markdown that no code names, reaches no test.
• Vitest and Jest: their own --changed and --changedSince.
• Anything it can't follow runs every test: the TypeScript or npm setup, test fixtures, a file deleted or renamed, a file no code names, very many files at once, or another runner.

The daily health run and release ready always run everything. Each case on the page says what it ran.

Set it up in your project

A Node project on node:test, Vitest or Jest without a test:affected script is offered one under Faster tests. Set it up makes a branch, assayer/affected-tests, from the project's branch in a worktree of its own, with the script line (and for node:test, a small script needing nothing installed). It's tested at once and opened as a ready pull request. Once merged, npm run test:affected runs only the tests your change reaches, on your PC or in CI.

Workspaces (npm, pnpm or Yarn workspaces, Nx, Turborepo) are offered nothing: their own tools do this across packages.