Skip to system research
Build A Trading SystemRules / Evidence / Replay

Frozen holdouts and the trading research change log

Keep selection decisions, rejected candidates and untouched evaluation periods auditable.

Original edition: 8 October 2026. Source limits appear below. Expanded edition: 13 October 2026.

Educational research design and invented arithmetic examples. No strategy returns, broker execution or production certification are supplied.

Define what is being selected

List the hypothesis family, candidate rules, parameter bounds, input transformations and selection objective before evaluating the reserved period. A search over entry thresholds, exits, universes and reporting windows is larger than a search over entry thresholds alone. Record the real choice set.

Separate economic ranking from software tests. A candidate that crashes should fail correctness checks. That does not make a test fixture an independent market period or demonstrate that a surviving strategy has an edge.

Give periods one declared job

Use historical development data to build the candidate and a reserved interval to evaluate the frozen choice. State how warmup crosses boundaries and how positions already open at a boundary are handled. Overlapping labels or holding periods can leak information even when rows are assigned different dates.

There is no universal percentage split or ideal sample size for every market. Dependence, horizon, regime coverage and the number of research decisions matter. Explain the actual design rather than attaching confidence to a convenient calendar division.

Freeze the candidate identity

Save code revision, full parameters, feature definitions, data identities, entry convention, exit policy, costs and acceptance rules. A short model nickname is not enough. Re-running with a corrected timezone or a new exit produces a different experiment even if the nickname stays the same.

Hashing a manifest can identify its bytes; it cannot establish that the declared holdout was never viewed. Research access and decision logs supply a different kind of evidence. Keep those claims distinct.

Record failure without reopening the tournament

A failed reserved test is information. Do not repeatedly adjust the candidate against that same interval while continuing to call it untouched. A new hypothesis can be researched, but its exposure to previous results must be acknowledged and its next evaluation designed accordingly.

Log discarded candidates and the reason for each change. A final report that shows only the winning specification hides selection risk. QuantConnect's research guide discusses overfitting and reserved evaluation; this site's manifest checklist is an editorial implementation of that concern, not a statistical certification.

Make the decision trace readable

For each experiment keep a question, predeclared settings, data period, result, known defects and next action. A bug fix may invalidate the original result, while a failed economic hypothesis may remain a valid negative finding. Do not merge those outcomes into a single attractive chart.

Before any operational use, reproduce the frozen candidate and compare orders with the intended policy. A verified no-deployment decision is preferable to promoting a model whose evidence cannot be reconstructed.

Keep access history beside the manifest

Record when each reserved result first became visible and which decisions had already been frozen. A file hash alone cannot establish that a researcher never viewed a period. Access records and a decision log provide a different evidential layer. If the history is incomplete, describe the evaluation as exposed or uncertain rather than silently restoring the untouched label.

Sources and scope

Next research step

Trading-system expectancy after costs

No untouched financial holdout is manufactured

The small reference has one fixed setting chosen for a software exercise and no optimiser. Its invented observations are not independent financial development and holdout periods. The packet says economic validation false. A future source change or parameter revision must preserve the old object, and a viewed real interval cannot regain untouched status through a renamed manifest. The new overlap and search-ledger investigations add specific evidence questions without reopening the original frozen-policy contract. Actual entry selection remains separate from exit-adjusted validation and requires the full authorised research/master process for a real instrument task.