Property-Based Testing (PBT)
Example-based tests miss edge cases — property-based testing with Hypothesis, fast-check, jqwik, QuickCheck, or proptest; invariants, round-trip, oracle and metamorphic properties, shrinking, stateful/model-based tests.
Instead of asserting f(3) == 9, assert a property that must hold for all inputs; the framework generates hundreds of cases, then shrinks any failure to a minimal counter-example.
Find properties (the hard part) — 7 patterns
- Round-trip:
decode(encode(x)) == x(serializers, parsers, compress, crypto). - Invariant:
len(sort(xs)) == len(xs), output is ordered, total money conserved. - Idempotence:
f(f(x)) == f(x)(normalize, dedupe, PUT). - Oracle: compare with a slow/simple reference implementation.
- Metamorphic:
search(q + filter) ⊆ search(q);price(2×qty) == 2×price(qty). - Commutativity/associativity where the domain promises it.
- No-crash / postcondition: for any valid input, no exception and result satisfies the contract.
Examples
# Python — Hypothesis
from hypothesis import given, strategies as st, settings
@given(st.lists(st.integers()))
def test_sort_props(xs):
out = sorted(xs)
assert len(out) == len(xs)
assert all(a <= b for a, b in zip(out, out[1:]))
assert sorted(out) == out # idempotent
// TypeScript — fast-check
import fc from "fast-check";
test("json round-trip", () => {
fc.assert(fc.property(fc.jsonValue(), v => expect(JSON.parse(JSON.stringify(v))).toEqual(v)));
});
pytest --hypothesis-seed=12345 # reproduce a failing run
npx jest --testTimeout=30000 # PBT can be slower
Generator design
- Constrain to the valid domain with strategies (
st.integers(0, 120)), notassume()filters that discard most inputs. - Build domain objects with composite strategies (
@st.composite,fc.record). Include nasty values: empty, huge, unicode, NaN, negative zero, duplicates. - Keep generators fast; set
max_examples(default 100) up in CI nightly (1000+), down for PR (50).
Stateful / model-based testing
Model the system as a simple state machine; generate random command sequences; compare real system to the model after each step (Hypothesis RuleBasedStateMachine, fast-check commands). Excellent for caches, queues, carts, DB wrappers, REST CRUD.
Reproducibility
Persist failing examples (Hypothesis .hypothesis/ DB; commit @example(...) for regression); log seeds in CI. Never mark a PBT flaky — a failure is a real counter-example (or a bad property).
Pitfalls
Re-implementing the function inside the property (tautology) · properties too weak (assert True) · non-determinism in the code under test · unbounded generators blowing runtime.
Related
unit-testing, fuzz-testing, test-design-techniques, mutation-testing