A category needs to survive another reader

A coding guide may seem clear to its author because the author already knows what each category is intended to mean. A second reader tests a different question: can the written instructions support a comparable interpretation without that private context? A published COSMIN study examined agreement and reliability when different raters applied its methodological checklist.[1] That study concerns the COSMIN checklist, not NSI. It provides an example of evaluating a rating process rather than assuming that a shared document automatically produces a shared judgment. Proposed NSI coding needs its own appropriately designed evaluation.

Save the independent versions before a conversation

For a hypothetical pilot, give two readers the same set of anonymised notes and the same version of a coding guide. Ask each to record a category and mark insufficient information when the note does not support a judgment. Save those versions before allowing discussion. Define the Measure Before Naming the Score explains why the categories need an explicit meaning. Score the Record Before Revealing the Outcome adds a separate control over which later information is visible. These are proposed workflow questions, not results from a completed NSI study.

Disagreement can identify a problem in the instructions

Imagine one note saying, “I felt rushed but followed the written entry rule.” One reader codes urgency; another codes rule adherence. The disagreement may reflect two different target questions rather than an error by either reader. Clarify whether the guide permits separate fields and what each field represents. Keep the original disagreement when revising the guide. A later consensus should not be reported as if it were the initial independent agreement. Avoid choosing a statistical agreement measure without qualified methodological advice; the design, category frequencies and intended interpretation matter to that choice.

Reliability does not answer every validity question

Even if readers apply a definition consistently, another question remains: does that definition support the interpretation or use proposed for the score? Do not turn agreement between coders into a claim of biological measurement or profitable prediction. NSI, Signals and the Confidence Trap connects these levels of evidence. NSI.GOLD and The Gold Nervous System Indicator introduce a proposed framework for studying the human trading decision. Independent assessment is work that could help evaluate it, not a substitute for validation. A useful pilot report preserves instructions, original codes, disagreements and revisions so that someone outside the project can understand what actually happened.

The Gold Nervous System Indicator by Gabriel Nicolaev – CyGuru
EXPLORE THE RELATED BOOK

The Gold Nervous System Indicator

Discover the connection between the human factor and gold trading.

Discover the book ↗Read the book overview →

Sources & further reading

  1. Mokkink and colleagues — Inter-rater agreement and reliability of the COSMIN checklist ↗

Sources consulted 9 November 2026. Editorial frameworks and illustrative examples are identified within the text.

CONNECTED PROJECTS

Take the next step.

NSI.GOLD ↗Explore the proposed framework and the independent measurement work needed to evaluate it.
More in Gold & Trading →Editorial principles