Testing Avelune
Validation layers and regression policy for the canonical implementation and independent oracle.
View authoritative sourceAvelune treats tests as evidence about meaningful codec properties, not as a line-counting exercise. A test should normally protect a format invariant, an implementation-independence contract, a state-machine property, a resource boundary, or a measured rate/quality/performance property.
Evidence layers
-
Normative unit and differential tests cover syntax, reconstruction, resource limits, and exact agreement between the canonical implementation and the independent Draft Generation 1 oracle.
-
Property tests generate valid values and frames with shrinking, targeting canonical varints, entropy round trips, lossless video/audio, CPU dispatch, and thread-policy invariance.
-
Metamorphic tests check properties that should survive representation changes: fragmentation, reserve/commit versus copy, epoch reset, and lossless decode/re-encode.
-
Stable mutation tests deliberately perturb valid codec/container seeds while requiring deterministic, panic-free classification.
-
Coverage-guided fuzzing uses separate entropy, ALV1, ALA1, container-streaming, and structured-valid-video targets. Structured generation is important because random malformed bytes otherwise spend most fuzz time in early rejection paths.
-
Stateful soak/concurrency tests exercise reference eviction, allocation reuse, epoch reset, independent streams, and independent decoder instances over long runs.
-
Sanitizers instrument the safe canonical engine and the isolated unsafe kernel crate on scheduled CI.
-
End-to-end integration gates exercise the CLI, container streaming, scalar/SIMD WASM, exact HTTP Range behavior, local Blob playback, browser encoding, and actual Chromium.
-
Deterministic codec regression tests build PR base and head on the same runner and encode the same generated media. Size and quality receive tight, tradeoff-aware checks; timing only has broad catastrophic-regression thresholds because hosted-runner timing is noisy.
-
Scheduled benchmark history records machine-readable JSON/CSV at multiple quantizers plus the implementation lab. Plotting and analysis are downstream concerns rather than part of the codec runtime.
-
Heterogeneous external media is an additional evidence tier, not a definition of the real world. Generated edge cases, adversarial inputs, fuzzing, and normative tests remain independent acceptance constraints.
PR versus scheduled CI
PR CI must remain fast enough that developers do not routinely skip it. Stable Rust property/metamorphic/mutation tests and a small deterministic media comparison therefore run on PRs. Nightly fuzzing, sanitizers, long soak tests, and repeated benchmark history are scheduled or manually triggered.
Performance regression policy
Deterministic properties such as output size and decoded quality can be compared tightly. Wall-clock measurements are noisy on shared CI hardware, so PR timing gates only reject large same-runner regressions. Raw samples and medians are retained in artifacts so scheduled or dedicated hardware can apply stricter statistical analysis later.
A size increase is not automatically a failure if it buys a meaningful quality gain, and a quality loss is not automatically a failure if it buys a meaningful rate reduction. CI records both instead of optimizing a single number.