Test like a race engineer.

Telemetry, configuration, controlled experiments and fast feedback make motorsport a useful analogy for serious engineering verification and validation.

Engineering testing illustration inspired by telemetry and race engineering

A race engineer does not improve a car by collecting random laps. Every run has a configuration, objective, environment and evidence set. If the driver reports instability, the team tries to reproduce the condition, isolates variables, checks telemetry and decides whether the symptom belongs to setup, hardware, software, tyres, track conditions or the test itself.

Testing is not “running the system.” It is a controlled experiment designed to answer a specific engineering question.

1. Start with the question, not the test case

Before creating steps, identify the claim you are trying to support. Verification normally asks whether a specified requirement is met. Validation asks whether the product is suitable for the stakeholder’s intended use and operational context.

Question	Reference	Typical evidence
Did we build it correctly?	requirements/specification	analysis, inspection, demonstration, test
Did we build the right thing?	stakeholder need / intended use	representative scenarios, user/field evaluation

2. Define acceptance before execution

If the pass criterion is decided after seeing the result, the test is already biased. A strong test case identifies preconditions, configuration, stimulus, expected result, tolerance, data to record and invalid-run rules before execution.

TC-142  Brake-response latency
Requirement: SYS-REQ-142
Configuration: ECU SW 3.12.4 / CAL EU_07
Stimulus: rising edge on request signal
Expected: actuator feedback <= 200 ms
Data: request_ts, response_ts, bus trace
Invalid if: dropped frame or sync error > 2 ms

3. Configuration control is part of the result

“The test passed” means very little if nobody can identify the exact software, calibration, hardware, environment and procedure that produced the pass. A race team records setup because a lap time without configuration is not reproducible. V&V needs the same discipline.

ProductHW / SW / calibration

Test systembench / model / instruments

Evidencelogs / traces / report IDs

4. Change one meaningful variable at a time

Controlled experiments help separate cause from correlation. If software, calibration, environment and test script all change at once, a better result tells you very little. The principle is simple even when the system is complex: reduce uncontrolled variation and document the factors you intentionally vary.

5. Failures need classification, not emotion

A red test result can mean at least four different things: product defect, test-script defect, test-environment failure or invalid data. Treating all failures as product bugs creates noise and destroys trust in the test campaign.

Observed failure	Investigation
Requirement not met	confirm configuration, reproduce, inspect implementation
Intermittent timeout	timestamps, load, network trace, race conditions
Unexpected numeric result	units, scaling, calibration, data pipeline
Test cannot execute	bench health, fixtures, tool versions, dependencies

6. Automate stable logic, not confusion

Automation is powerful when the test basis is mature. A bad manual procedure turned into Python simply becomes a fast bad procedure. I prefer separating instrument drivers, reusable fixtures, domain calculations and acceptance assertions.

@pytest.mark.parametrize("profile", reference_profiles)
def test_gap_algorithm(profile):
    result = run_gap(profile.data)
    assert result.status == "valid"
    assert abs(result.gap - profile.reference_gap) <= profile.tolerance
    assert result.evidence.transition_points is not None

The last assertion is deliberate: for engineering algorithms, sometimes the evidence object deserves tests too.

7. Traceability closes the loop

A professional V&V workflow connects stakeholder need → requirement → verification method → procedure → result → anomaly → change → regression evidence. Traceability is not bureaucracy when used properly; it is how you answer “what is affected if this requirement changes?”

Need→Requirement→Test→Evidence→Decision

8. Regression testing is an impact-analysis problem

Running every test after every change is often impossible. Running only the test for the changed function is often unsafe. Regression selection should use architecture, interfaces, requirement links, defect history and risk to determine what could have been affected indirectly.

9. Validation needs representative context

A system can pass every laboratory requirement and still disappoint in real use. Validation asks whether the operational scenarios, users, environment, workload and failure modes are represented well enough to support the final claim. This is where “works on my bench” stops being an engineering argument.

10. A test report should support a decision

The output of a campaign is not a pile of screenshots. A useful report identifies scope, configuration, requirements covered, methods, results, open anomalies, deviations, limitations and unresolved evidence gaps. The reader should be able to understand what is known, what is not known and what decision the evidence supports.

11. Test architecture should separate orchestration from measurement logic

In automated engineering tests, hardware communication, stimulus generation, measurement, domain calculations and acceptance assertions should not be one monolithic script. Separation makes the campaign easier to maintain and makes failures easier to diagnose.

driver  -> acquires signals
fixture -> prepares known state
model   -> derives engineering quantities
assert  -> evaluates acceptance criteria
report  -> stores evidence + configuration

12. Confidence comes from orthogonal evidence

The strongest verification argument rarely depends on one test technique. Reviews expose requirement defects, analysis checks limits, static inspection finds structural problems, simulation explores scenarios cheaply, and physical tests confirm behaviour in representative environments. Independent evidence reduces the chance that one blind spot becomes the final answer.