Skip to system research
Build A Trading SystemRules / Evidence / Replay

Common trading-system mistakes and the record that catches them

Recognise hidden research choices, denominator errors, timing leakage and policy rewrites through concrete counterexamples.

Editorial revision and linked-document check: 10 October 2026. Document-based analysis; no paid-account test or independently reproduced provider return.

The final indicator list is not the whole search

A system's universe, observation clock, cost model, reporting interval and exit policy can all change its result. Logging only the final threshold hides much of the research process. Save material choices and abandoned runs with the information already inspected when each decision was made.

An attractive chart produced after many experiments is not equivalent to one predeclared test. This guide is an error checklist, not a survey measuring how often traders make each mistake. Its examples are constructed to reveal reasoning problems without supplying invented performance or practitioner success statistics.

A library split does not solve the entire experiment

The scikit-learn documentation explains fitting and evaluation boundaries, including ordered splitting for time-dependent observations. Using a recognised splitter does not automatically prevent leakage through revised inputs, overlapping outcomes or a transformation fit before the split. Inspect those boundaries separately.

The backtest-overfitting research by Bailey and co-authors concerns strategy selection across trials. A citation does not turn an undocumented search into a measured selection-adjusted result. Record the complete candidate set and evaluation design before claiming an independently reproduced probability or significance statement.

A deliberately misleading win-count example

In an invented ledger, one winning idea reaches three targets while one losing idea stops out. Counting the targets as three wins and the loss as one trade produces 75%. Counting the two ideas gives a 50% winning-idea rate. Neither establishes monetary profit without quantities and gain or loss sizes.

Choose the unit before computing the rate, then apply it consistently to favourable and adverse events. Partial exits belong to the same idea with their remaining exposure. The counterexample is not an observation about a named seller; it shows how a plausible headline can arise from a mismatched denominator.

Timestamp repair is not new independent evidence

Suppose a fictional replay used a completed bar before its actual completion. Correcting the order timing can materially change the curve. Preserve both the defective run and the repaired specification, explain the defect and rerun all dependent checks. Do not relabel the original attractive result as valid.

A passing unit fixture establishes the repaired calculation for that case. If the revised strategy was chosen after inspecting its outcomes, the same observations are not untouched confirmation. Keep software correctness, developmental selection and independent economic evaluation in separate receipts so a bug fix cannot hide their different roles.

Quiet policy changes alter the subject

Changing an exit, an ambiguity convention or a missing-data rule changes the experiment even when the entry list looks identical. Give the new policy a new identity and retain the original outcomes. Avoid favourable per-trade exceptions invented after viewing the path. Administrative closures also need a declared rule.

A production label is not deployment evidence. Distinguish research state, merged code, registry admission, deployed version, subscriber route and certification. A file called live can be useful research material while the customer-facing system still serves an older configuration or no configuration at all.

Make review possible without a story

An independent reviewer should receive the specification, source fingerprints, candidate log, event ledger, gate outputs and known omissions. They should be able to reproduce the selected interpretation without relying on the author's memory of why an exception was allowed. A negative or no-model result should remain part of the record.

The blank checklist is a way to locate defects, not an automatic quality score. No number of completed fields certifies an economic edge. State what is verified, what is rejected and what remains unimplemented, then route the next experiment through the same declared standards instead of weakening the gates to rescue a favourite result.

Build a Trading System evidence objects
Keep the publication's evidence objects separate when assembling your review.

Evidence handoff

Three checks to retain with this assessment
CheckRecordBoundary
Hidden choiceCandidate and decision logFinal settings omit the search
Miscounted outcomeIdea identity and partial quantitiesTargets are not independent ideas
Repaired implementationOriginal defect and dependent rerunsA fixture is not a strategy sample
Blank evidence CSV

Column definitions: issue_id, experiment_id, candidate_count, data_access_date, timing_convention, outcome_unit, cost_policy, entry_version, exit_version, original_run, corrected_run, reserved_data_used, deployment_evidence, remaining_check.

No example trade or user record is stored in the download. Keep credentials and personal account identifiers out of shared copies.

A reference identity catches a changed story

The public fixture is not searched for a positive result. Its original negative 1.20 remains beside the source, data, settings and transactions. Replacing a cost, altering a future bar or changing an exit produces another object even if the same nickname is retained. The checksum inspector can identify the changed bytes but cannot prove why they changed or whether an evaluation was inspected. A new hash is not a fresh holdout. The full prior method and source gaps remain necessary before interpreting a financial claim, rather than using a clean software demonstration to hide selection or accounting choices.