Write a trading-system data-availability contract
Define what each feature could know at decision time before running the backtest.
Keep selection decisions, rejected candidates and untouched evaluation periods auditable.
Educational research design and invented arithmetic examples. No strategy returns, broker execution or production certification are supplied.
List the hypothesis family, candidate rules, parameter bounds, input transformations and selection objective before evaluating the reserved period. A search over entry thresholds, exits, universes and reporting windows is larger than a search over entry thresholds alone. Record the real choice set.
Separate economic ranking from software tests. A candidate that crashes should fail correctness checks. That does not make a test fixture an independent market period or demonstrate that a surviving strategy has an edge.
Use historical development data to build the candidate and a reserved interval to evaluate the frozen choice. State how warmup crosses boundaries and how positions already open at a boundary are handled. Overlapping labels or holding periods can leak information even when rows are assigned different dates.
There is no universal percentage split or ideal sample size for every market. Dependence, horizon, regime coverage and the number of research decisions matter. Explain the actual design rather than attaching confidence to a convenient calendar division.
Save code revision, full parameters, feature definitions, data identities, entry convention, exit policy, costs and acceptance rules. A short model nickname is not enough. Re-running with a corrected timezone or a new exit produces a different experiment even if the nickname stays the same.
Hashing a manifest can identify its bytes; it cannot establish that the declared holdout was never viewed. Research access and decision logs supply a different kind of evidence. Keep those claims distinct.
A failed reserved test is information. Do not repeatedly adjust the candidate against that same interval while continuing to call it untouched. A new hypothesis can be researched, but its exposure to previous results must be acknowledged and its next evaluation designed accordingly.
Log discarded candidates and the reason for each change. A final report that shows only the winning specification hides selection risk. QuantConnect's research guide discusses overfitting and reserved evaluation; this site's manifest checklist is an editorial implementation of that concern, not a statistical certification.
For each experiment keep a question, predeclared settings, data period, result, known defects and next action. A bug fix may invalidate the original result, while a failed economic hypothesis may remain a valid negative finding. Do not merge those outcomes into a single attractive chart.
Before any operational use, reproduce the frozen candidate and compare orders with the intended policy. A verified no-deployment decision is preferable to promoting a model whose evidence cannot be reconstructed.
Record when each reserved result first became visible and which decisions had already been frozen. A file hash alone cannot establish that a researcher never viewed a period. Access records and a decision log provide a different evidential layer. If the history is incomplete, describe the evaluation as exposed or uncertain rather than silently restoring the untouched label.
Trading-system expectancy after costs
Define what each feature could know at decision time before running the backtest.
Design idempotency, partial fills, stale signals and restart reconciliation before live execution.
The small reference has one fixed setting chosen for a software exercise and no optimiser. Its invented observations are not independent financial development and holdout periods. The packet says economic validation false. A future source change or parameter revision must preserve the old object, and a viewed real interval cannot regain untouched status through a renamed manifest. The new overlap and search-ledger investigations add specific evidence questions without reopening the original frozen-policy contract. Actual entry selection remains separate from exit-adjusted validation and requires the full authorised research/master process for a real instrument task.