A microbiome classifier can be tested by turning some measured values into zeros and asking whether its predictions still hold up. Before running that test, someone has to decide which microbial features to leave untouched.
The public benchmark I examined protected features associated with the outcome—in this case, a diabetes label. If that selection uses the labels of the very samples being evaluated, the answer can enter the construction of the test even when it never enters classifier fitting. I built a comparison to separate that contribution from the other differences between protection rules.
Why the earlier comparison was not enough
I first compared protection selected from the complete dataset with protection re-estimated inside training folds. Its repeated cross-fitting results across six datasets are in the v0.2.0 preprint released on 31 August 2026.
That comparison changed more than access to test labels. The selectors had different numbers of observations available, selected different features, and could respond differently to the split. A performance gap did not isolate the contribution of the realised pairing between test rows and their outcomes. For the next version, I built a control that kept the full data available while breaking only that pairing during feature selection.
Changing the label pairing, not the whole experiment
The main contrast uses two protection rules. Arm A selects protected features from all rows and their actual labels. Arm B uses those same rows and leaves training labels unchanged, but shuffles the test labels within the test fold before selecting the mask. The fold retains the same number of positive labels; only their assignment to particular test rows is broken.
This is the paired sham-label control. Both arms are still scored against the original test labels. Shuffling occurs only in the mask-selection step.
A and B share the fitted classifier, test rows, protected-set size and requested amount of corruption. The difference Δ(A−B) therefore asks how much the realised test row–label pairing contributes under that selector and perturbation scheme. It is not a comparison between two separately fitted models.
I kept other controls because they answer different questions. Arm C selects features using outer-training data only. Arm D protects a fixed-size random set without using outcomes. Synthetic runs also include arm E, which protects prespecified spike coordinates first; its meaning is interpreted only in scenarios where signal was actually added. A−C measures the change from full-data to training-only selection; it should not be treated as interchangeable with A−B.
Testing a setting with no predictive relationship to recover
Before interpreting a relationship between diabetes and microbes, I used conditions where the labels carried no predictive relationship. One experiment generated synthetic abundance data independently of the labels. Another kept each observed feature matrix intact and randomly reassigned its full label vector. That reassigned vector becomes the scoring target for the null experiment; arm B then makes a further within-test-fold shuffle only to select its protection mask. Removing the original predictive relationship and constructing the A–B control are two separate steps.
The synthetic experiment covered two dimension designs, three signal conditions and two model families, with 200 repetitions in each of 12 cells: 2,400 dataset–model runs. The fixed-matrix null experiment used two datasets and two model families, with 200 label permutations per cell: 800 runs. These are repeated computational experiments on fixed data, not 800 newly recruited participants. Confirmatory analyses used only the previously unused repetition indices 1000–1199; pilot runs were excluded.
All four complete-null synthetic cells had a positive mean A−B AUROC difference. In the WT2D-sized design, it was 0.1554 for random forest and 0.0690 for XGBoost; in the T2D-sized design, the corresponding values were 0.1203 and 0.0718. AUROC measures ranking discrimination between two classes. These differences are not improvements in clinical diagnostic accuracy.
The matrix-preserving null-label controls showed the same direction.
| Fixed feature matrix | Model | Mean A−B AUROC |
|---|---|---|
| WT2D | Random forest | 0.0578 |
| WT2D | XGBoost | 0.0213 |
| T2D | Random forest | 0.0318 |
| T2D | XGBoost | 0.0122 |
The prespecified primary WT2D/random-forest contrast had a 95% bootstrap interval of 0.0527–0.0630. The other three cells were checked with the prespecified multiplicity correction. The effects differed in size, so the results do not provide a universal amount to subtract from a robustness score.
The secondary synthetic scenarios added genuine signal: 1 or 2 log units in the first 8 absolute-abundance coordinates of the WT2D-sized design, or the first 12 in the T2D-sized design. Stronger signals improved recovery of those coordinates, but A−B differences remained and varied across conditions. These scenarios examine the coexistence of predictive signal and an evaluation-construction effect; they do not model a biological effect size. Development analyses using the observed diabetes labels were excluded from the confirmatory results.
Holding the model and amount of change fixed
To interpret those score differences, I had to check what else changed alongside the protected features. WT2D has 96 samples and 381 microbial features, with 18 protected features. T2D has 344 samples and 572 features, with 36 protected. I analysed the matrices separately.
For evaluation, I shuffled each dataset and divided it into five groups with similar diabetes-label proportions. Each group took a turn as the test set while the others supplied the training data: five-fold shuffled stratified cross-validation. Both random forest and XGBoost used 100 trees. Within a fold, I fitted the model once and reused it across all protection arms. Refitting a model could itself change its predictions, so that variation was kept out of the paired comparison.
Checking how many entries actually became zero
For the outcome-based masks, ANOVA statistics ranked features by differences associated with the diabetes label. Unprotected columns underwent a power transformation, which drove small values lower. Values below 10⁻⁶ were set to zero, and each row was renormalised.
I also fixed how far this process should go. Sparsity here means the fraction of matrix entries that are zero. The target was the fourth of five interior grid points between the starting zero fraction and 0.99, giving about 0.909 and 0.937 for the two observed matrices.
A shared target alone would not be enough if one mask left fewer entries available to change. I recorded both that capacity and the number of zeros actually added, checking that the realised dose stayed within one matrix entry of the requested dose. For the performance score, I gathered every sample's prediction from a model that had not trained on that sample. AUROC was calculated from these complete out-of-fold predictions, not by averaging the five fold-level AUROCs.
What the uncertainty describes
I did not judge the effect from one generated dataset or one shuffled label vector. An independent synthetic repetition or label permutation was the Monte Carlo unit. Resampling these repetition-level results 10,000 times gave a bootstrap interval for the mean at the 95% level.
The prespecified primary empirical contrast used a one-sided test for a positive A−B effect. The other three combinations formed a replication family with Holm correction, accounting for the possibility of finding a difference when making several comparisons. These intervals describe variation conditional on the fixed matrices and algorithms. They do not measure uncertainty for a newly sampled patient population.
What the public package lets a reader check
Version 0.3 includes the locked protocol, code, derived metrics, prediction and dose records, and a separately implemented verifier. The stored verification report records 81 passing checks and an exact deterministic replay of 36 synthetic tasks. Its empirical checks reconstruct AUROCs and summaries from stored probabilities; they do not refit every empirical model from scratch.
The finding concerns evaluation construction in two microbiome matrices, two tree-based model families, and one zero-inflation mechanism at one severity. It does not establish biological validity, clinical benefit or misconduct by the source authors. For an audit of a single realised dataset, repeated sham assignments within each fixed split would also be needed to distinguish sham-permutation variation from split variation.
The earlier v0.2.0 preprint contains the initial repeated cross-fitting study. I released the paired sham-label design and confirmatory null experiments on 3 September 2026 in the v0.3 public manuscript and its code and results package. The verifier is linked at the same fixed commit. Raw upstream abundance matrices are not redistributed; their source commit and file hashes are recorded. The earlier preprint and v0.3 public manuscript are not journal versions of record.