04 · Software Engineering
Testing & Quality — what tests are for, and how reviewers think
"How do you test your code?" is rarely about pytest syntax. It probes whether you understand risk — where bugs come from, what's worth defending, and whether you can be trusted with a shared codebase. Your Sentinel work gives you a rare angle: you built a system that reviews code and gates changes, so you've thought about quality from both sides.
01First principles: why do tests exist?
Not to prove correctness — that's impossible for nontrivial programs. Tests exist to:
A test is a tripwire you pay for once and it guards forever.
The real payoff isn't finding today's bug — it's letting you refactor next quarter without fear. A codebase without tests calcifies because nobody dares touch it.
A good test name is a spec that can't drift stale: test_expired_token_returns_401 tells you more than a wiki page.
Hard-to-test code is nearly always badly-coupled code. (More in §7.)
02The testing pyramid
/ E2E \ few — slow, flaky, high fidelity
/ integr. \ some — real DB/queue, seams verified
/ unit \ many — fast, isolated, preciseThe shape encodes a cost argument:
| Layer | What it verifies | Cost profile | What it catches |
|---|---|---|---|
| Unit (base) | A function in isolation — pure input → output | Milliseconds each; thousands per commit; pinpoint failures to a function | Logic bugs, edge cases in parsers, pricing, state machines |
| Integration (middle) | Seams — your code against a real Postgres, a real Kafka topic, actual FastAPI routing + serialization | Slower, but tolerable with containers | Bugs unit tests structurally cannot see: SQL vs. schema mismatch, serialization, transaction semantics, config wiring |
| E2E (tip) | The deployed system, driven like a user | Slow, expensive, flaky (network, timing, shared environments) | Whole-system wiring; keep a handful of critical-path smoke tests, not hundreds |
What each layer looks like concretely (FastAPI world)
The function that classifies a log line as incident/noise — pure input → output, no I/O.
TestClient / httpx.AsyncClient against your app with a real (containerized) Postgres — assert the endpoint writes the row, returns 201, enforces the unique constraint.
Staging deploy, real webhook fired at it, assert a PR appears. Run nightly / pre-release, not per-commit.
03Mocking philosophy — the part people get wrong
A test double replaces a real dependency. Using the vocabulary precisely reads senior:
| Double | What it is | You assert on… | Example |
|---|---|---|---|
| Stub | Returns canned data; you don't assert on it | Your code's output, not the stub | "The user service returns this user." |
| Mock | You assert on how it was called | Call count + arguments | "Assert we called send_email once with X." |
| Fake | A real, working, lightweight implementation | Behavior through the fake | In-memory repository; SQLite standing in for Postgres. |
| Spy | Records calls for later inspection without changing behavior | The recording, afterwards | "Did the retry loop fire 3 times?" |
Mock: third-party HTTP APIs (GitHub, LLM providers), the clock, randomness, email/SMS, payment providers — slow, non-deterministic, cost money, or have side effects.
Don't mock: your own domain logic, your own repository layer in a test that exists to verify persistence, pure functions.
If your test mocks five internal collaborators to test one class, the test is a change-detector: it verifies your implementation calls your implementation, passes even when behavior is wrong, and fails whenever you refactor. This is the London school vs Detroit school debate — mockist vs classicist — and the pragmatic position is classicist-leaning: assert on observable behavior (return values, state changes, external effects), not on internal call sequences.
Two failure smells to name in interviews
30 lines of mock setup, 2 lines of assertion — and the assertion is mock.assert_called_once(). It tests the mocks.
The mock's shape no longer matches the real dependency, so tests pass and prod fails. Mitigations: autospec=True in unittest.mock (mocks enforce the real signature), contract tests, or recorded-cassette tools (VCR.py / respx) that pin real response shapes.
04TDD, honestly assessed
The loop: red (write a failing test) → green (minimal code to pass) → refactor (clean up under a green bar). Repeat in small steps.
What it genuinely delivers
• Forces you to define behavior before implementation — you catch fuzzy requirements at the cheapest moment.
• Guarantees the test can fail (you watched it fail) — a surprising number of non-TDD tests pass vacuously.
• Produces testable, decoupled design as a side effect, because you experience the API as a caller first.
• Great for bug fixes: write the failing repro test first, so the bug can never silently return.
• Great for logic-dense, well-specified problems: parsers, validators, pricing rules, state machines.
Where it honestly fits poorly
• Exploratory work — when you don't know what the interface should be yet (prototyping an agent loop, trying a chunking strategy), test-first ossifies your first guess. Spike without tests; throw the spike away or backfill tests once the shape settles.
• Glue-heavy code — thin controllers, config wiring: TDD ceremony exceeds value; an integration test covers it better.
• Systems whose "correctness" is empirical — LLM output quality, ranking relevance, UI feel. You need evals/experiments, not red-green-refactor.
05Property-based testing — the idea
Example-based test: for this input, expect this output. Property-based test: for all inputs (sampled), this invariant holds. The framework (Hypothesis in Python) generates hundreds of randomized inputs and — crucially — shrinks any failure to a minimal counterexample.
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_sort_is_idempotent(xs):
assert sorted(sorted(xs)) == sorted(xs)
@given(st.text())
def test_roundtrip(s):
assert decode(encode(s)) == s # the killer pattern: encode/decode round-tripsClassic property families
serialize→deserialize = identity (Pydantic models, chunking→reassembly, event encode/decode).
Output sorted, no items lost, total conserved.
Fast clever implementation agrees with slow obvious one.
f(f(x)) == f(x) — normalizers, your webhook dedupe logic.
"\x00", surrogates, off-by-one boundaries. You don't need to claim daily use; knowing when you'd reach for it (parsers, codecs, anything with an invariant) is the signal.06pytest patterns that read senior
Fixtures — dependency injection for tests
@pytest.fixture
def db(postgres_container): # fixtures compose
engine = create_engine(postgres_container.url)
run_migrations(engine)
yield engine # test runs here
engine.dispose() # teardown after yield
@pytest.fixture
def client(db):
app.dependency_overrides[get_db] = lambda: db # FastAPI's DI hook
return TestClient(app)Key points: yield gives setup/teardown in one place; scopes (function default, module, session) trade isolation for speed — container per session, clean transaction per test (begin → run test → rollback) is the classic fast-and-isolated DB pattern; conftest.py shares fixtures without imports.
Parametrize — one test, a table of cases
@pytest.mark.parametrize("status,expected", [
(200, "ok"), (429, "backoff"), (500, "retry"), (400, "fail_fast"),
])
def test_retry_policy(status, expected):
assert classify(status) == expectedEach row is a separately-reported test. This is how you cover status-code matrices, permission matrices, and edge tables without copy-paste.
Others worth having in your pocket
pytest.raises(ValueError, match="negative") # assert exceptions precisely
@pytest.mark.slow # markers → tiered suites:
pytest -m "not slow" # fast per-commit, slow pre-mergeFixture for env vars / attributes; freezegun or an injected clock for time.
pytest-asyncio / anyio for async handlers; respx for mocking httpx calls (your LLM-provider tests).
Quarantine-and-fix, never normalize re-running. A suite people re-run is a suite people ignore.
test_duplicate_webhook_delivery_is_ignored). If a test needs a paragraph of comments, the name is wrong.07What "testable code" actually means
Testability isn't a testing property — it's a design property. Code is testable when:
def process(event, *, github: GitHubClient, clock: Clock) — a test passes fakes. Code that does requests.post(...) or datetime.now() inline welds itself to the world.
"Functional core, imperative shell." This one pattern is 80% of testability — see below.
Functions return results instead of mutating globals/singletons; hidden module-level state makes tests order-dependent (the worst flake source).
An interface/protocol for "notifier", "repo store", "LLM provider" means each has a fake.
Time, randomness, UUIDs, network — all injected.
Welded to the world
def handle(event):
logs = requests.get(LOG_URL).json() # hidden I/O
if datetime.now().hour < 6: # hidden clock
return
action = decide(logs, event["diff"])
slack.post(action.summary) # hidden effectFunctional core, imperative shell
# core: pure decision fn — fast, exhaustive unit tests
def decide_fix_action(logs, diff, now) -> Action: ...
# shell: thin fetching/posting layer —
# a few integration tests cover it
def handle(event, *, http, slack, clock):
action = decide_fix_action(http.logs(), event.diff, clock.now())
slack.post(action.summary)08Code review culture — what reviewers actually look for
You've built an automated adversarial reviewer, so speak about review with unusual authority: you had to encode "what makes a change trustworthy" into a system.
The reviewer's priority stack (roughly in order)
Does this change do what the ticket needed? Is the approach right, or is it a local patch on a design problem?
Nulls/empties, concurrency, partial failure (what if the third of five writes fails?), idempotency of retried paths, timezone/encoding traps.
Injection (SQL/command/prompt), authz on new endpoints (who can't call this?), secrets in code/logs, unsafe deserialization, SSRF on URL-fetching features.
Migration safety (backwards compatible? locking a hot table?), rollout/rollback story, feature flags, what appears in logs/metrics when this fails at 3am.
Do tests cover the behavior that changed, and would they fail if the change were reverted? 400 lines of code with tests that never exercise the new branch is the classic red flag.
Naming, function size, whether the next person can understand it. Consistency with the codebase beats personal taste.
Formatting belongs to Black/Ruff, not humans. Teams that argue about commas in review waste their scarcest resource: reviewer attention.
Culture markers of good teams (and good candidates)
Review quality collapses with diff size — a 2,000-line PR gets "LGTM"; a 200-line PR gets found bugs. Authors who split work reviewably are prized.
Self-review your own diff, annotate the non-obvious bits ("this looks redundant but handles the redelivery race") — respect for reviewer time.
"What happens if the queue is down here?" > "wrong." Prefix nits as nit: so severity is legible.
As author, treat pushback as free QA, not attack. Verify feedback technically rather than performatively agreeing — some review comments are wrong, and saying so with evidence is healthy.
Reviews spread codebase knowledge as much as they find defects — it's how juniors absorb the codebase and how bus-factor stays >1.
09Rapid-fire interview Q&A
Q: How do you decide what to test?
Q: Is 100% coverage a good goal?
Q: How do you test code that calls an LLM?
Q: How do you handle a flaky test?
Q: Unit or integration for a CRUD endpoint?
10Self-test before you close the tab
Name the four test doubles and the one-line difference between mock and spy.
Stub (canned data, no assertions on it), mock (assert on how it was called), fake (real lightweight implementation), spy (records calls for later inspection without changing behavior). Mock = you set the expectation up front and assert the interaction; spy = it just records, and you inspect the recording afterwards.
Give the pyramid's cost argument in two sentences, then the trophy counter-argument.
Unit tests are cheap, fast, and pinpoint failures, so you want many; E2E tests are slow, expensive, and flaky, so you want few — the shape is an economic discipline. Trophy counter: on service-heavy backends most real bugs live at the seams, and testcontainers/in-process clients make integration tests fast enough to be the workhorse layer.
What's the classicist objection to heavy mocking, and what do you assert instead?
Over-mocked tests are change-detectors — they verify your implementation calls your implementation, pass when behavior is wrong, and fail on every refactor. Assert on observable behavior instead: return values, state changes, external effects — and mock only at boundaries you don't own.
Scope TDD honestly: two places it shines, two where it fits poorly.
Shines: bug fixes (failing repro test first) and logic-dense, well-specified code (parsers, validators, pricing, state machines). Fits poorly: exploratory work where the interface is unknown (spike, then backfill) and empirical-correctness systems (LLM output quality, ranking) that need evals, not red-green-refactor.
Recite the reviewer priority stack from memory.
1 correctness of intent → 2 edge cases & failure modes → 3 security → 4 blast radius & operability → 5 tests that cover the changed behavior → 6 readability & maintainability → 7 style/nits, last and automated.
Why does "functional core, imperative shell" buy you 80% of testability?
Pure decision functions (logic) get fast, exhaustive, deterministic unit tests with no mocks; the thin shell that fetches and posts gets a few integration tests. Non-determinism (time, network, randomness) is pushed to the edges and injected, so nothing needs heroic patching.