MRI-002 Historical Holdout Summary

Status: frozen holdout validation; no method tuning was performed.
Inputs remained local/offline. Existing historical master and calibration files were not modified.

Eligibility
- Source eligible high-engagement population: historical/mri_high_engagement_discussions.csv
- Calibration exclusions: historical/mri_predictive_120_discussions.csv
- Calibration articleIDs in holdout: 0
- Holdout discussions: 813

Matching Validation
- Future winners: 8130
- Target matched controls: 24390
- Actual matched controls: 24390
- Median absolute order-percentile difference: 0.001757
- 90th percentile order-percentile difference: 0.004175
- Controls within +/-0.01: 97.79%
- Controls within +/-0.025: 99.70%
- Controls within +/-0.05: 100.00%
- Duplicate/reused controls within article: 0
- Controls with comment_order <= 25: 0
- Controls that are FUTURE_TOP_10 winners: 0

Missing Data
- Candidate rows missing text: 0
- Candidate rows missing timestamps: 0
- Candidate rows missing recommendation counts: 0
- Candidate rows missing early-alignment similarity: 0

Frozen Text Alignment Method
- Text sources used for representation: EARLY_25 text and blinded matched candidate text only.
- Cleaning: HTML tags and URLs removed; text lowercased.
- TF-IDF: English stop words, word unigrams+bigrams, min_df=2, sublinear_tf=True, max_features=50000.
- TruncatedSVD: n_components=100, random_state=42.
- Vectors normalized before cosine similarity.
- Semantic representation was created before candidate roles were revealed for metric calculation.
- Control selection used comment_order_pct distance, createDate, and commentID only; it did not use recommendations or text.

Holdout Early-Alignment Metrics
- Pairwise early-alignment win rate: 0.613571
- Winner > mean-control rate: 0.651415
- Top-of-four accuracy: 0.362485
- Chance baseline for top-of-four accuracy: 0.250000
- Fraction of holdout discussions with mean winner-minus-control similarity > 0: 0.859779

Length Robustness
- Conditional logistic regression groups: matched pair_id sets.
- Model: winner_status ~ standardized_early_alignment_similarity + standardized_log1p_word_count.
- Fitting method: statsmodels ConditionalLogit
- standardized_early_alignment_similarity: coefficient=0.135104; odds_ratio=1.144656; 95% CI [1.103629, 1.187208]; p=4.0253e-13
- standardized_log1p_word_count: coefficient=0.607256; odds_ratio=1.835388; 95% CI [1.767831, 1.905526]; p=4.84793e-221

Compact Outputs
- historical/mri_holdout_summary.txt
- historical/mri_holdout_discussion_results.csv