The Eventide Project is happy to announce the third generation of TestBench, the principal test framework used in the Eventide stack and by the Eventide community.

What’s Changed?

This is a major release that includes new features, minor breaking changes, and the restoration of two features that had been removed in Gen 2.

Breaking Changes

• Stricter assertions: assert and refute no longer accept nil, only true or false.
• Fixture modules removed: If you were using fixture modules (not classes), those are gone. Fixture classes are still here.
• Session substitute cleanup: Several undocumented predicate method aliases have been removed.
• Block handling in fixture: If you pass a block to fixture, it now always goes to the fixture’s constructor. No more magic invocation fallback.
• New environment variable: TEST_BENCH_ONLY_FAILURE is replaced by TEST_BENCH_OUTPUT_LEVEL, with four modes: abort, failure, not-passing, and all.
• Comments and details: comment and detail no longer accept a variadic list of arguments.

Restored Features

• Backtrace filtering: Use TEST_BENCH_OMIT_BACKTRACE_PATTERN with glob patterns (not regex) to ignore irrelevant frames.
  • Relative paths: Backtraces also use relative paths when possible. Easier to read.
• Abort on first failure: TEST_BENCH_ABORT_ON_FAILURE stops the batch runner as soon as one file fails.

New Features

• Pseudorandom generator: TestBench::Random gives you random strings, integers, decimals, and booleans. Deterministic if you set TEST_BENCH_RANDOM_SEED.
• Strict mode: With TEST_BENCH_STRICT set to on, skipped tests fail the run, and at least one test must be executed.
• Comment improvements:
  • You can now pass fixtures to comment or detail, and their output will be printed.
  • Comments and details have an optional heading parameter as the first positional argument.
  • Optional style keyword parameter with five styles: block, line_number, normal, indent, and raw
• Context colorization: Context titles now reflect result status—red for aborts, green for passes, and unstyled if no tests ran.
• Output control:
  • Choose where output goes with TEST_BENCH_OUTPUT_DEVICE (stdout, stderr, or null).
  • Disable the final summary by setting TEST_RUN_OUTPUT_SUMMARY to off.
• Test files now run in isolated subprocesses. If a test file crashes, it’s replaced with a new, pristine process.

For Gen 2 Users

You’ll want to test your current projects against the prerelease. Most projects won’t need to change much—unless you were relying on internals or undocumented experimental features.

Final Word

This generation was about honing and fine tuning TestBench, having gathered a list of incremental improvements over several years of practical application.

Please report any issues or concerns to the #test-bench channel in the Eventide Project’s Slack organization.