Three expression values, 2, 4 and 6, add up to 12. Subtract their mean first and they become −2, 0 and 2, which add up to zero. Starting with 4, 8 and 12 leads to the same zero sum. These are illustrative numbers, not experimental measurements, but they capture the problem I found while checking a comparison function in IRIS: preprocessing was removing the information the score was meant to add.
IRIS uses RNA measurements from individual cells to infer signaling states and histories. Nicholas Hutchins and colleagues trained a neural network on signaling perturbation experiments and evaluated its transfer to other cellular contexts. Their paper appeared in Nature Methods on 8 September 2026. I began with a simpler part of the comparison: the response-gene baseline used alongside the neural network.
What the baseline was supposed to measure
A response-gene score adds the expression of genes known to respond to a particular signal. Here, expression is first normalized for the amount of RNA measured in each cell and then log-transformed. This gives a simple reference against which to judge what a more complex model contributes.
The Methods description and the released response_gene() function performed different calculations. The function first subtracted the mean of the selected genes within each cell and divided by their standard deviation, then added the resulting z-scores. Deviations from the same mean sum to zero; dividing them by a common standard deviation preserves that zero sum. Floating-point arithmetic can leave small residuals, but those residuals cannot serve as a meaningful measure of signaling response.
The three-number example isolates that cancellation. The actual calculation involves pathway-specific gene sets, per-cell normalization, logarithms and gene weights. My initial task was to establish where the released calculation departed from the written baseline, before drawing conclusions about the neural network compared with it.
From a discrepancy report to a checked correction
I reported the discrepancy to the original authors. On 24 September, Nicholas Hutchins confirmed that the function had been corrected and supplied the location of the pretrained checkpoints. The correction itself is recorded in a public commit dated 19 September UTC. My contribution was identifying and checking the discrepancy; Hutchins made the upstream correction. His commit explanation states that the corrected function matches the existing resp_calc() calculation used to produce the published figures.
The change removes both the within-cell z-scoring and a final division by the maximum score in the supplied cell subset. Gene weights remain, with a default of one, so the default score is the intended sum of normalized, log-transformed expression. Removing the maximum division also avoids changing a cell’s score scale according to which other cells happen to be included in the same call.
On 27 September, I inspected the old and corrected implementations and executed the corrected function on four synthetic integer-count rows. Its maximum difference from a separately calculated version of the declared sum was 0.0. That checks this function against its intended formula. It does not reproduce every published figure or the complete neural-network analysis.
Taking a fixed model into a different dataset
Once a trained model was available, I also ran a bounded transfer test using an independent human neural organoid screen. Organoids are cultured cell systems that reproduce some features of developing tissue. This screen included retinoic-acid (RA) exposure and control conditions, allowing a test of whether the released RA classifier separated the recorded exposure histories. The neural organoid setting differs from the model’s training context.
I fixed the protocol on 24 September, before examining real-cell predictions. The panel contained 32 cells from each combination of two experimental batches, four cell lines, two media and three conditions: 1,536 cells in total, with 512 controls and 512 in each of RA_A and RA_E. The 27 September run used one released gastrulation RA checkpoint without new training or threshold tuning.
Matching inputs exposed a practical limit. Of the model’s 13,222 features, 13,078 matched current gene symbols and another 88 matched unique previous symbols. The remaining 56 were unmeasured. I had specified zero filling for this engineering test, but an unmeasured gene is not a demonstrated biological zero. That input policy was my adaptation, not an author-validated procedure for applying IRIS to external data.
The pooled comparison was only part of the result
I compared the fixed classifier with the prespecified sum of nine RA response genes. AUROC summarizes how well a score ranks exposed cells above controls: roughly, the chance that a randomly selected exposed cell receives a higher score than a randomly selected control, with ties accounted for. A value of 0.8 is not a claim that 80% of cells were classified correctly.
| Fixed comparison | AUROC across 1,536 cells |
|---|---|
| IRIS RA classifier, with unmeasured inputs set to zero | 0.7994 |
| Sum of nine RA response genes | 0.8311 |
The simple baseline had the higher pooled value. Within the 16 matched batch–medium–cell-line groups, however, IRIS had higher AUROC in nine and the baseline in seven. The pooled result did not support a claim that the baseline was uniformly better.
Two prespecified stress tests filled all 56 unmeasured inputs with one count or ten counts. These were artificial counterfactuals, not estimates of the missing expression or bounds on biologically plausible values. Relative to zero filling, they changed 80 classifications (5.21%) and 225 classifications (14.65%), respectively, at the fixed 0.5 decision threshold.
The 1,536 cells were nested observations from culture pools and two experimental batches, not 1,536 independent experiments. I did not calculate cell-level significance tests or confidence intervals. Recorded RA exposure was the reference label, rather than a direct measurement of current intracellular signaling activity; differences in cell composition could also affect separation. The exact dose and timing represented by RA_A and RA_E remained unresolved in this work.
What I could close, and what remains open
The discrepancy in the released baseline function could be closed with an upstream correction and a bounded execution check. The separate transfer test records how an explicit missing-input policy affected predictions, and why the pooled comparison needed to be read alongside matched conditions.
I did not turn that transfer result into a refutation of the paper’s biological conclusions. The original study already reports its own validation and transfer evaluations, and my adapted input handling does not reproduce those methods exactly. A stronger assessment would first need supported handling of the 56 missing inputs and an exact interpretation of the RA conditions. I preserved the completed calculation and left further interpretation contingent on resolving those input and exposure details.
Sources: IRIS paper · official repository · response_gene correction · pretrained checkpoints · external organoid study. OpenAI Codex assisted with analysis code, reproducibility checks, and writing.