NSI Research: Keep the Test Set Outside the Workshop

A result can become familiar before it is called a test

Suppose a proposed NSI score is developed on a collection of trading observations. The researcher then checks a later period, dislikes one outcome, adjusts a threshold and checks that same period again. The later period may retain its label as a test set even though it has joined the development process. The issue is the information flow, not the researcher’s sincerity. NSI is discussed here as a proposed framework requiring evaluation. Neither the name of the framework nor a carefully designed worksheet establishes physiological validity, predictive ability or a trading advantage.

Draw the boundary before looking across it

Kapoor and Narayanan identify data leakage as a source of overoptimistic results in machine-learning-based science.^[1] Their paper is methodological context, not a study validating NSI. Apply the lesson through a concrete separation: define the observations used to develop the method and the observations reserved for its evaluation. Keep a dated account of who accessed each set and why. Writing the Test Before Seeing the Outcome covers the prior commitment; this article concerns the boundary that must survive afterwards. If outcomes influence feature definitions or thresholds, that influence belongs in the method’s history.

Follow one hypothetical revision honestly

Imagine a rule developed on an initial period and inspected on a later period. A researcher changes the definition of high stress after seeing that later performance. That revision may be worth investigating, but the inspected period no longer provides an untouched assessment of the revised rule. Label the result exploratory, preserve both versions and plan an evaluation on new eligible observations. Do not delete the awkward original result. The same principle applies when preprocessing or exclusions are chosen because they improve an outcome. There is no numerical performance claim in this example; it illustrates how a procedure can lose its independence.

Separate different kinds of evidence

A separation between development and evaluation is necessary discipline, not a complete validation programme. Data quality, participant selection, measurement reliability and the relevance of the comparison remain separate questions. Read The Observations That Never Reach the Journal before treating a clean spreadsheet as a representative sample. If a score uses personal-state reports, those reports are not automatically clinical measurements. If it uses market observations, association is not by itself a decision rule that survives costs. The Gold Nervous System Indicator book describes the connected framework; the book and the NSI project should be read alongside independent methods literature, rather than cited as independent confirmation of themselves.