Bellweather Plugin Testing Suite (BPTS)
Every plugin passes 100% of our test suite before release. No exceptions. Here's what we test and why it matters.
How This Connects Across The Site
Testing Suite is the methodology layer. Observatory is the operational execution layer. Research Notes are the evidence layer with plugin-specific outcomes, known gaps, and mutation-testing findings.
What is BPTS?
BPTS (Bellweather Plugin Testing Suite) is our custom testing framework built on industry-standard tools: Catch2 for unit tests via CTest, Pluginval for VST validation, and our own BOL (Bellweather Observation Lab) for characterization testing.
We don't just test if a plugin works. We test if it survives the worst-case scenarios: rapid parameter changes, extreme values, thread race conditions, and intentionally injected bugs.
Release Lanes
Every commit runs through automated release lanes in CI/CD. Each lane gates progressively more of the test suite:
fast
Whitebox fast subset only. Runs on every PR to catch obvious regressions.
gate
Full whitebox regression set. Catch2 via CTest with bpts_gate labels. Must pass before merging to main.
rc
Headless-safe release candidate gate. Whitebox tests (slow on), Pluginval when GUI available (SKIPs with reason=no_gui_session in headless), and BOL gate.
rc_gui
GUI-required release candidate gate. Same as rc but Pluginval is mandatory—FAILs if no GUI session is available.
lab
Extended whitebox + BOL lab characterization. Transfer curves, THD analysis, and envelope response verification against committed baselines.
What runs per lane
| Lane | Whitebox Gate | Pluginval | BOL Gate | BOL Lab |
|---|---|---|---|---|
| fast | optional (fast) | — | — | — |
| gate | yes | — | — | — |
| rc | yes (slow on) | yes / SKIP headless | yes | — |
| rc_gui | yes (slow on) | required (FAIL if no GUI) | yes | — |
| lab | yes (slow on) | — | — | yes |
Core Test Types
Category 5 Stress Tests →
25 failure scenarios every plugin must survive. RT-safety, parameter mutations, memory leaks, and artifact detection.
The Gauntlet™ →
1 million simultaneous parameter changes. If a plugin survives this, it will survive anything your DAW throws at it.
Mutation Testing →
Intentionally inject bugs to verify our tests actually work. How we find blind spots in our testing.
BOL Characterization →
Bellweather Observation Lab. Transfer curves, THD analysis, and envelope timing verification.
Why This Matters
Most plugin companies don't publish their testing methodology. We do because transparency matters more than appearing perfect.
Every test in BPTS exists because something broke in the past. We learned the hard way that you can't ship quality without rigorous testing.
If you find a bug our tests missed, we want to know. That's how we improve the framework for future plugins.
Status Semantics
- PASS
- Step executed and succeeded.
- SKIP
- Step intentionally not run (e.g., Pluginval with no GUI in
rc). - FAIL
- Step executed and failed, or required precondition missing (e.g.,
rc_guiwithout GUI).