Black-box rule of the handbook

Regla de caja negra del manual

This handbook does not redefine the case, rediscover variables, or decide which evidence is needed. Its task is to execute the selected method architecture, calculate all required indices, run the necessary diagnostics, quantify uncertainty, and produce a standardized technical evidence object for Module IV.

Este manual no redefine el caso, no redescubre variables ni decide qué evidencia se necesita. Su tarea es ejecutar la arquitectura metodológica seleccionada, calcular todos los índices requeridos, correr los diagnósticos necesarios, cuantificar incertidumbre y producir un objeto técnico estandarizado para el Módulo IV.

1. Position in the full framework

1. Posición dentro del framework completo

MODULE I
Case and variable eligibility
        ↓
MODULE II
Evidence requirements
        ↓
MODULE III — Conceptual Layer
Method capability and method selection
        ↓
MODULE III — Method Validation Handbook
Execution + indices + diagnostics + uncertainty
        ↓
METHOD EVIDENCE REPORT
        ↓
MODULE IV
Interpretation + decision rules + acceptance logic
        ↓
PASS / CONDITIONAL / FAIL / RETURN

2. Handbook objective

2. Objetivo del manual

The objective is to convert each selected method into a reproducible quantitative execution package. Every package must specify inputs, preprocessing, parameters, algorithm execution, indices, diagnostics, uncertainty, sensitivity, robustness, failure signals, and output artifacts.

El objetivo es convertir cada método seleccionado en un paquete cuantitativo reproducible de ejecución. Cada paquete debe especificar entradas, preparación, parámetros, ejecución algorítmica, índices, diagnósticos, incertidumbre, sensibilidad, robustez, señales de fallo y artefactos de salida.

3. Universal validation pipeline

3. Pipeline universal de validación

1. Receive the ESS requirement and selected MSS method.
2. Recibir el requisito ESS y el método seleccionado en el MSS.
3. Resolve the exact algorithmic variant and parameter space.
4. Resolver la variante algorítmica exacta y el espacio de parámetros.
5. Prepare the data using only transformations allowed by the ESS and MSS.
6. Preparar los datos usando solamente transformaciones permitidas por ESS y MSS.
7. Execute the primary method and all required companion procedures.
8. Ejecutar el método principal y todos los procedimientos complementarios requeridos.
9. Calculate native indices, effect measures, uncertainty, and diagnostics.
10. Calcular índices nativos, medidas de efecto, incertidumbre y diagnósticos.
11. Run sensitivity, robustness, resampling, and failure-detection procedures.
12. Ejecutar sensibilidad, robustez, remuestreo y detección de fallos.
13. Generate tables, plots, residual objects, logs, and machine-readable outputs.
14. Generar tablas, gráficos, residuos, logs y salidas legibles por máquina.
15. Assemble the Method Evidence Report without contextual acceptance decisions.
16. Ensamblar el Method Evidence Report sin decisiones contextuales de aceptación.

4. Universal method execution specification

4. Especificación universal de ejecución del método

METHOD EXECUTION SPECIFICATION

Case ID:
ESS Evidence ID:
MSS Method ID:
Evidence Type:
Scientific Question:

Primary Method:
Exact Algorithm / Variant:
Software / Library / Version:
Random Seed:
Execution Environment:

Input Variables:
Target Variable:
Unit of Analysis:
Grouping Structure:
Time Structure:
Spatial Structure:

Required Preprocessing:
Excluded Transformations:
Missing-Value Strategy:
Scaling / Encoding:
Feature Construction:
Leakage Controls:

Hyperparameters / Parameters:
Search Space:
Selection Rule:
Stopping Rule:

Required Companion Methods:
Required Baseline:
Required Resampling:
Required Sensitivity Tests:
Expected Output Objects:

5. Universal index architecture

5. Arquitectura universal de índices

Every method package must calculate indices in distinct families. The exact indices vary by method, but the families remain constant.

Cada paquete metodológico debe calcular índices en familias distintas. Los índices exactos varían por método, pero las familias permanecen constantes.

6. Universal diagnostics layer

6. Capa universal de diagnósticos

7. Sensitivity analysis protocol

7. Protocolo de análisis de sensibilidad

Sensitivity analysis determines how much the calculated evidence changes when defensible analytical choices change.

El análisis de sensibilidad determina cuánto cambia la evidencia calculada cuando cambian decisiones analíticas defendibles.

8. Robustness protocol

8. Protocolo de robustez

Robustness analysis evaluates whether the method evidence remains materially similar under plausible disturbances to the data or execution environment.

El análisis de robustez evalúa si la evidencia metodológica permanece materialmente similar bajo perturbaciones plausibles de los datos o del entorno de ejecución.

9. Evidence-family validation registry

9. Registro de validación por familia de evidencia

10. Example: prediction method package

10. Ejemplo: paquete de método predictivo

EVIDENCE TYPE: PREDICTION
PRIMARY METHOD: Random Forest

CALCULATE
- Accuracy
- Precision
- Recall / Sensitivity
- Specificity
- F1
- ROC-AUC
- PR-AUC
- Log Loss
- Brier Score
- Calibration Slope
- Calibration Intercept
- OOB Error
- Confusion Matrix
- Permutation Importance
- Importance Stability

DIAGNOSTICS
- Leakage Check
- Class Imbalance
- Group / Time Split Validity
- Calibration Curve
- Learning Curve
- Fold Variance
- Seed Variance
- Subgroup Performance
- Drift Sensitivity

REPORT
- Values
- Intervals
- Fold Distribution
- Plots
- Runtime
- Memory
- Failure Flags

DO NOT DECIDE HERE
- Whether AUC is sufficient
- Whether calibration is acceptable
- Whether the model should be deployed

11. Example: temporal method package

11. Ejemplo: paquete de método temporal

EVIDENCE TYPE: TEMPORAL PRECEDENCE
PRIMARY METHOD: Granger

CALCULATE
- Lag-Specific F Statistics
- p-values
- AIC
- BIC
- Residual Variance
- Reverse-Direction Test
- Forecast Improvement
- Confidence Intervals

DIAGNOSTICS
- Event-Time Validation
- Stationarity Checks
- Lag Selection
- Residual Autocorrelation
- Ljung–Box
- Stability Across Windows
- Out-of-Time Validation
- Common-Driver Sensitivity

REPORT
- Results by Lag
- Direction X→Y
- Direction Y→X
- Stable Lag Set
- Residual Diagnostics
- Sensitivity Tables
- Interpretation Boundary

DO NOT DECIDE HERE
- Whether precedence is causal
- Whether one significant lag is sufficient
- Whether the result supports intervention

12. Example: causal method package

12. Ejemplo: paquete de método causal

EVIDENCE TYPE: POLICY EFFECT
PRIMARY METHOD: Difference in Differences

CALCULATE
- Treatment Effect
- Standard Error
- Confidence Interval
- Event-Study Coefficients
- Pre-Treatment Coefficients
- Cluster-Robust Uncertainty
- Subgroup Effects

DIAGNOSTICS
- Parallel-Trend Evidence
- No-Anticipation Check
- Spillover Review
- Composition Stability
- Placebo Dates
- Placebo Outcomes
- Alternative Control Groups
- Alternative Time Windows

REPORT
- Main Effect
- Event-Study Plot
- Pre-Trend Table
- Placebo Results
- Sensitivity Results
- Population Scope
- Identification Assumptions

DO NOT DECIDE HERE
- Whether the causal claim is accepted
- Whether the effect is practically important
- Whether the policy should be adopted

13. Failure detection framework

13. Framework de detección de fallos

14. Method Evidence Report (MER)

14. Method Evidence Report (MER)

METHOD EVIDENCE REPORT

Case ID:
ESS Evidence ID:
MSS Method ID:
Execution ID:
Timestamp:
Software / Version:
Data Snapshot ID:

SCIENTIFIC CONFIGURATION
Evidence Type:
Scientific Question:
Inputs:
Target:
Unit of Analysis:
Temporal / Spatial / Group Structure:

METHOD EXECUTION
Primary Method:
Exact Variant:
Parameters:
Preprocessing:
Baseline:
Companion Methods:
Resampling:
Sensitivity Plan:

RESULTS
Primary Estimate:
Magnitude Indices:
Uncertainty:
Fit Indices:
Generalization Indices:
Calibration Indices:
Residual Diagnostics:
Stability Indices:
Robustness Results:
Computational Profile:

ARTIFACTS
Tables:
Plots:
Residual Objects:
Model Object:
Logs:
Machine-Readable Output:

FAILURE FLAGS
Execution:
Input:
Diagnostics:
Stability:
Support:
Reproducibility:

INTERPRETATION BOUNDARY
Allowed Interpretation:
Prohibited Interpretation:

MODULE IV HANDOFF
Indices Ready:
Diagnostics Ready:
Sensitivity Ready:
Robustness Ready:
Decision Status:
NOT EVALUATED IN MODULE III

15. Module III validation statuses

15. Estados de validación del Módulo III

16. Generic workshop

16. Taller genérico

17. Final handoff contract to Module IV

17. Contrato final de entrega al Módulo IV

MODULE IV RECEIVES

- What method was executed
- Why it was selected
- Exact configuration
- Exact data used
- All primary indices
- All uncertainty measures
- All diagnostics
- All sensitivity results
- All robustness results
- All failure flags
- All plots and tables
- All interpretation boundaries
- No hidden thresholds
- No hidden exclusions
- No pre-decided acceptance

Handbook closing

Cierre del manual

The Method Validation Handbook is the quantitative engine of Module III. It transforms a conceptual method choice into a reproducible execution, a complete set of indices, and a traceable Method Evidence Report. It does not decide whether the evidence is sufficient for the case. It ensures that Module IV receives everything required to make that decision transparently and consistently.

El Method Validation Handbook es el motor cuantitativo del Módulo III. Transforma una selección conceptual en ejecución reproducible, un conjunto completo de índices y un Method Evidence Report trazable. No decide si la evidencia es suficiente para el caso. Garantiza que el Módulo IV reciba todo lo necesario para decidir de forma transparente y consistente.