Two scores came out of a test of how well a model could predict lipophilicity, a measure of how strongly a molecule prefers an oily environment to water. Both scores came from the same public dataset and the same kind of model. About 0.50 was the average of 20 runs that used one way of representing the molecules and kept groups of similar molecules together when assigning them to training or testing. About 0.70 was the highest score across 120 runs that varied how the data were split and how molecules were represented. Both numbers are R² scores. The higher the score, the more of the variation in measurements the model explains in that test set.
Three ways to choose the test set
I ran the comparison during the November 13–15 Apart Research AI × Epistemics sprint. I used ESOL, FreeSolv and Lipophilicity, three public datasets of measured molecular properties. I compared splitting the molecules at random, keeping molecules with the same core structure together, and keeping groups of structurally similar molecules together.
For each dataset I combined the three splitting methods with two ways of representing molecules and 20 random seeds. A seed fixes the random choices for a run. I trained and tested a random forest for each combination, for a total of 360 runs.
Across the three datasets, the best single score exceeded the best 20-run average with a similarity-grouped split by 0.059 to 0.205 on the R² scale. I changed the evaluation setup and picked a single run, so the gap shows how far the reported number could move across the choices I tried. The comparison covers three datasets and one model family.
What I check before trusting a score
Before training a model, I can check how familiar the test material is. For each test molecule, I find its closest match among the training molecules and look at how similar they are. This helps me see which comparison I’m actually asking the model to make.
