Published molecular ML methods, re-run on the same molecules, the same folds and the same statistics.
Python
1
25 commits
updated Oct 18, 2026
Published machine learning methods for molecular property prediction, re-run on the same molecules, the same folds and the same statistics, so their claims can be read against one another instead of against whatever each paper chose to compare itself with.
Most studies here take a published method, or a published claim about a class of methods, and put it through one fixed evaluation protocol on two unrelated ADME data sets. The methods differ. The molecules, the splits, the endpoints and the statistics do not.
One does not, and is marked below. When a paper's claim is about the splits, that protocol cannot test it, because holding the split fixed is the thing being questioned. Such a study runs on the source paper's own molecules and its own folds instead, and carries a reproduction of one of that paper's arms as the evidence the two sets of numbers belong on the same axis. It is comparable within itself and not with the cards above it.
Every study is scored the same way, and that is the point of collecting them here.
GroupKFold
over the clusters in what remains. Every method sees identical training molecules in
every fold and is scored on the same untouched test set, so a difference in the
metrics is a difference in the method.Four reference arms — LightGBM on Morgan counts, a single-task D-MPNN, a multitask
D-MPNN, and a D-MPNN initialised from CheMeleon — are fit once, in
studies/expansion-ml-comparison, and their per-fold predictions are carried into the
other ADME studies rather than refit, so those cards all sit on one axis. The
cross-study table at the top of the index pools only the studies that carry those
arms; site/overview.py refuses anything else, which is why the split-stringency
study does not appear in it.
Does pre-training a 3D transformer on quantum-mechanical properties help ADMET prediction?
QuADMET-Former pre-trains a 1.3M-parameter TorchMD-ET equivariant transformer on PubChemQC quantum properties — total energy, HOMO–LUMO gap, dipole moment, E3FP fingerprints and Löwdin charges — across ten released checkpoints, then adapts it by linear probing followed by fine-tuning. Both a fine-tuned arm and its frozen embedding read by TabPFN 3 are run on the authors' code and weights, against LightGBM, fine-tuned CheMeleon, Monroe + TabPFN 3 and fine-tuned Atom-JEPA.
What came out. Pre-trained LP-FT beats from-scratch training on 14 of 15 endpoints by a median +0.083 R², and LightGBM on 38 of 45 combinations. Against the pre-trained field it reaches the top Tukey tier on 5 of 27 ExpansionRx combinations, nearly splits with fine-tuned CheMeleon (15 wins, 9 ties, 21 losses over both data sets), and trails fine-tuned Atom-JEPA (2 wins, 35 losses) and Monroe + TabPFN 3 (10 wins, 33 losses).
Puts to the test
Six discrepancies between the paper and the authors' repository were found and are documented in the study README; the arms here use the released checkpoints with those issues worked around.
6 methods · 15 endpoints · 10 released QM checkpoints · 1.2 h GPU time
Code: studies/quadmet-former · Report builder: 08_build_page.py
Does a 3D equivariant encoder pre-trained on 19 million conformers beat the best frozen encoder?
Atom-JEPA pre-trains a 39M-parameter EquiformerV3 by latent prediction on 19M Uni-Mol conformers and fine-tunes it, multitask, on ten conformers per molecule. It is run here with the authors' own code and released weights, against the arm the paper did not run, Monroe + TabPFN 3, and a frozen Atom-JEPA arm read by the same TabPFN 3 call separates the representation from the fine-tuning.
What came out. Fine-tuned Atom-JEPA is ahead of fine-tuned CheMeleon on 38 of 45 combinations and of LightGBM on all 45. Against Monroe it is ahead on 6 of 9 ExpansionRx endpoints and behind on 4 of 6 Biogen ones, and training-set size does not explain the split (ρ = −0.04). Frozen, its embedding never beats Monroe's, a median −0.09 R²; fine-tuning never loses to freezing and adds +0.13 on ExpansionRx and +0.07 on Biogen.
Puts to the test
7 arms · 15 endpoints · 50 fine-tuned fold models · 46 h GPU time
Code: studies/atom-jepa · Report builder: 08_build_page.py
Does correcting each prediction with the known errors of similar training molecules help?
MolPAIR corrects one tabular foundation model with another: a global TabPFN 3 reads a frozen CheMeleon fingerprint, and a second TabPFN 3 learns from pairs of training molecules how the first model's error changes between similar molecules, so each test molecule borrows the errors of up to 32 neighbours. It is put beside LightGBM, fine-tuned CheMeleon and Monroe + TabPFN 3.
What came out. Against the prediction it corrects, MolPAIR is significantly ahead on 41 of 45 endpoint × metric combinations and behind on 2, and improves 51–66% of individual predictions, close to the paper's 55–57%. The gain holds, at half the size, on the test molecules the paper's own split would have kept. But the correction is worth +0.024 R², fine-tuning CheMeleon +0.047 and Monroe's embedding +0.129: MolPAIR loses to fine-tuned CheMeleon on 30 combinations and to Monroe + TabPFN 3 on 34.
Puts to the test
A reimplementation. The paper's code link does not resolve, so this is written from its Section 3 and Appendix A.
6 arms · 15 endpoints · 375 fold models · 41/45 ahead of its baseline
Code: studies/molpair · Report builder: 06_build_page.py
Holding Monroe's frozen embedding fixed, which in-context tabular model reads it best?
Every tabular foundation model in NVIDIA's structured-data-models package — TabICLv2, Kumo Tabular small, medium and large, and the 1.6-billion-parameter TabFM — reads the same frozen 720-d Monroe embedding on the same folds as Monroe + TabPFN 3, so the only thing that differs between arms is the head. The authors' own tabicl package, loading the same checkpoint as NVIDIA's TabICLv2 port, is the control.
What came out. Kumo Tabular large is on top of all 45 endpoint × metric combinations after Tukey's correction, the only arm never significantly worse than the best. In paired tests it is ahead of TabPFN 3 on 28 and behind on 6, and ahead of TabFM on 28 against 7 for an eighth of the GPU time. NVIDIA's TabICLv2 port reproduces upstream tabicl to within 0.008 R². Swapping heads moves R² by a median of 0.042 per endpoint, against 0.208 for the representation.
Puts to the test
9 arms · 5 heads fitted · 15 endpoints · 45/45 Kumo large on top
Code: studies/sdm-tabular · Report builder: 05_build_page.py
A paper says nearest-neighbour lookup matches modern ML under every split, and that all of them collapse to a naive baseline out of distribution. Does that hold against models it did not have?
Eleven arms on the ten ChEMBL potency sets and six splitting schemes of Lee, Moldagulov and Grzybowski, run on that paper's own folds: four methods fitted here, the paper's naive baseline, and the paper's own six arms imported at their published per-fold numbers. 1,500 fold models, and a re-implementation of the paper's k-NN that reproduces 250 of 250 of its released folds to 4e-9, which is what licenses putting the two sources on one axis.
What came out. Half the paper replicates. Error does climb as the splits harden, but so does the naive baseline it is measured against, and reading the gap instead of the raw MAE splits the two claims apart. k-NN is at the top of none of the 18 scheme x metric combinations and Monroe's frozen encoder is at the top of all 18, separating from k-NN on all six schemes and on five of six under a conservative target-level test. On the hardest schemes k-NN takes 0.005 and 0.010 pIC50 off the naive predictor and the paper's own GNN goes negative, while Monroe holds 0.121 and 0.141. The decay rates are nearly identical across arms, 0.81 to 0.98; what differs is the offset, so nothing here is more robust to distance, it is just uniformly better.
Puts to the test
The paper's six arms are its own published per-fold metrics, not re-run, and each had 50 Optuna trials per fold where the four arms fitted here use library defaults. The upstream release ships two disagreeing copies of the splits; the .pkl copy is used, because it reproduces the paper's numbers and the .json copy does not.
11 arms · 6 splitting schemes · 1,500 fold models fitted · 250/250 published folds reproduced · 0/18 combinations k-NN leads
Code: studies/split-fragility · Report builder: 09_build_page.py
Which foundation model, and does it replicate?
Nine modelling approaches over fifteen ADME and physicochemical endpoints on two unrelated data sets, 25 replicate models each, every one scored on a held-out test set it never saw. Four of the nine are pre-trained molecular foundation models, and only one of those four wins anything at all. The one that wins does no downstream training whatsoever — it freezes a pre-trained encoder and predicts each endpoint in context — and it appears three times, under three different tabular models, which is what lets the page price the representation against the predictor bolted onto it.
What came out. Monroe's frozen encoder is at the top of 21 of 27 ExpansionRx combinations and all 18 Biogen ones; MEGA-CL and Mol-JEPA are best on nothing, anywhere. Crossing two frozen representations with two in-context heads settles which half does the work: changing the head moves R² by a median of 0.016 and not even in a consistent direction, while changing the representation moves it by 0.168 on fifteen endpoints out of fifteen. The representation is worth about ten times the head.
Puts to the test
9 methods · 15 endpoints · 2,925 fold models · 3.19M predictions kept
Code: studies/expansion-ml-comparison · Report builder: 06_build_page.py
A graph network taught to predict fingerprints, against the fingerprints
PT-GIN is a Graph Isomorphism Network pre-trained on 462,189 QMugs molecules to predict each molecule's own 2,048-bit hashed ECFP4, then frozen and handed to LightGBM. Its authors report it beating hashed ECFP on five of six Biogen ADME tasks. Here it is run against those same fingerprints through that same predictor, so the only thing that differs between the two arms is the representation.
What came out. It wins six endpoints of fifteen and loses six; on Biogen, its own benchmark, it wins one and loses three. Which six it wins is not about how much data an endpoint has — it is about how well the fingerprints were already doing. PT-GIN helps where the baseline was weak and hurts where it was strong, Spearman ρ = −0.65 with the baseline R².
Puts to the test
5 methods · 15 endpoints · 1,425 fold models · 10 released checkpoints
Code: studies/ecfp-pretrain · Report builder: 06_build_page.py
Unimodal versus multimodal, on ADME data
Thirty-three ways of combining four molecular representations — RDKit descriptors, Mol2Vec, a supervised GNN's graph embedding and a character BiGRU's SMILES embedding — crossed with early and late fusion and three final learners, over the same fifteen endpoints. 12,375 fitted configurations and 6,375 more for the two controls.
What came out. The paper's central finding replicates: fusing modalities moves accuracy very little, and none of it survives a correction for multiple comparisons. What fusion buys is calibration. Two things the paper could not see from inside its own design also show up — a single well-initialised graph network beats all thirty-three configurations on both data sets, and when the GNN modality is actually a learned representation, which in the released code it is not, the ranking of which modality matters inverts.
Puts to the test
A reimplementation. The authors' MIT-licensed release is a set of Colab dumps that cannot be run, so this is written from that source read as a specification alongside the paper's Supporting Information.
33 configurations · 15 endpoints · 18,750 fits · 4 modalities
Code: studies/multimodal-fusion · Report builder: 12_build_page.py
Is picking a model per endpoint worth it?
Trimole-Hybrid is not a model but a selection procedure. For every endpoint and every fold it builds sixty candidate predictors — five molecular views crossed with three blocks of chemistry priors crossed with four backends — fits all of them, scores them on the held-out validation fifth, and keeps exactly one to predict the test set. It is put against four fixed architectures on the same fifteen endpoints.
What came out. It loses to a single well-initialised graph network on both data sets, for two reasons only visible in what it selected: nothing in its pool learns a representation, and on ExpansionRx the split it selects on does not resemble the split it is scored on. What it chose does not replicate either.
Puts to the test
A reimplementation. The authors' release describes itself as not runnable, so this reproduces the method, not the paper's numbers.
5 methods · 60 candidates per fold · 22k candidate fits · 1,425 fold models
Code: studies/expansion-ml-comparison · Report builder: 14_build_trimole_page.py
studies/<slug>/ one self-contained study: scripts, README, requirements,
per-fold metrics, tables, figures, and a study.json
site/build_site.py reads every study.json, writes docs/
site/overview.py the cross-study Tukey comparison at the top of the index
site/style.py the stylesheet the index shares with the reports
docs/ the GitHub Pages site
docs/index.html the overall report, one card per study report
docs/reports/<slug>.html each study's own report, copied from its results/
docs/overview.json the tally table, and docs/assets/ the figures with it
A study directory stands on its own. It carries its own config.py,
requirements.txt, README and results, and it is run from inside itself, so it works
exactly as it did before it was collected here and can be lifted back out.
The index needs nothing but a Python interpreter:
python site/build_site.py # rebuild docs/ and the card block above
python site/build_site.py --check # say what is stale, change nothing
That copies each study's built report into docs/reports/ and regenerates the index
and the card block in this README from the study.json files. It does not run any
science. A study's report is built inside that study's own directory, in the
environment that study needs — see its README for the commands and its
requirements.txt for the environment.
The cross-study comparison at the top of the index is the one part that does read the numbers, so it is a separate script and needs pandas, statsmodels and matplotlib:
python site/overview.py # -> docs/overview.json, docs/assets/*.png
python site/build_site.py # embeds what it wrote
The table and the figures use the same metric, so a row and its panel can never
disagree about who won: one Tukey HSD per endpoint, counted best alone / tied /
worse. That metric is MAE by default, since it is in the units of the endpoint;
--metric r2 or --metric spearman rebuilds both against either of the others.
Run it when a study's fold_metrics.csv changes. It refuses to pool anything
unless the four reference arms are identical across the studies that carry them,
and it asserts the tie rule on every endpoint × metric combination before
counting.
Drop the study directory under studies/<slug>/. It keeps its own README,
requirements.txt and results.
Add a studies/<slug>/study.json:
{
"slug": "my-study",
"name": "my-study",
"datasets": ["ExpansionRx", "Biogen ADME"],
"endpoints": 15,
"protocol": "5 x 5 GroupKFold over chemical clusters, one fixed holdout, Tukey HSD",
"updated": "2026-09-02",
"reports": [
{
"slug": "my-study",
"date": "2026-09-02",
"title": "The title on the card and the report",
"eyebrow": "What kind of question this is",
"question": "The one-line question the study answers",
"summary": "Two or three sentences on what was compared and how much of it.",
"finding": "What came out, in one or two sentences, including the number that matters.",
"facts": [{"n": "15", "label": "endpoints"}],
"validates": [
{
"name": "Short name for the method",
"authors": "Author, A.; Author, B.; et al.",
"title": "The paper's full title",
"venue": "J. Chem. Inf. Model. 2026",
"label": "doi:10.1021/...",
"url": "https://doi.org/..."
}
],
"caveat": "Optional. Say so here if this is a reimplementation.",
"source": "results/report.html",
"builder": "06_build_page.py"
}
]
}
slug, title, question, summary, finding and source are required;
everything else is optional. A study with two reports — as
expansion-ml-comparison has — lists both, and gets a card for each.
Cards are ordered newest first, by each report's date, so a new study lands
at the top on its own and nothing has to be renumbered. A report without a
date falls back to its study's updated, and one with neither sorts last.
Reports built on the same day are separated by their study's slug, which only
keeps the order stable and means nothing.
Each entry in validates is a paper the study puts to the test, cited in
full on the card: title, authors and venue are printed, label is the
link text and url is where it points.
Run python site/build_site.py.
If the new study reuses the reference arms, point its importer at
studies/expansion-ml-comparison; studies/ecfp-pretrain/00_import_baselines.py is
the worked example.
Code is MIT — see LICENSE. Two data files travel with the studies that use them and keep their own terms:
expansion_log_scaled.csv — the OpenADMET–ExpansionRx blind challenge data,
contributed by Expansion Therapeutics, released by OpenADMET under CC BY 4.0
(doi:10.57967/hf/9687), and put on a
log10(x + 1) scale here. Please credit both.biogen_adme_3521.csv — built from
molecularinformatics/Computational-ADME,
released by Biogen under the MIT License with the paper of Fang et al.No third-party pre-trained weights are vendored. Each study's README says where its checkpoints come from and under what licence.
Published molecular ML methods, re-run on the same molecules, the same folds and the same statistics.
Python
1
25 commits
updated Oct 18, 2026
Published machine learning methods for molecular property prediction, re-run on the same molecules, the same folds and the same statistics, so their claims can be read against one another instead of against whatever each paper chose to compare itself with.
Most studies here take a published method, or a published claim about a class of methods, and put it through one fixed evaluation protocol on two unrelated ADME data sets. The methods differ. The molecules, the splits, the endpoints and the statistics do not.
One does not, and is marked below. When a paper's claim is about the splits, that protocol cannot test it, because holding the split fixed is the thing being questioned. Such a study runs on the source paper's own molecules and its own folds instead, and carries a reproduction of one of that paper's arms as the evidence the two sets of numbers belong on the same axis. It is comparable within itself and not with the cards above it.
Every study is scored the same way, and that is the point of collecting them here.
GroupKFold
over the clusters in what remains. Every method sees identical training molecules in
every fold and is scored on the same untouched test set, so a difference in the
metrics is a difference in the method.Four reference arms — LightGBM on Morgan counts, a single-task D-MPNN, a multitask
D-MPNN, and a D-MPNN initialised from CheMeleon — are fit once, in
studies/expansion-ml-comparison, and their per-fold predictions are carried into the
other ADME studies rather than refit, so those cards all sit on one axis. The
cross-study table at the top of the index pools only the studies that carry those
arms; site/overview.py refuses anything else, which is why the split-stringency
study does not appear in it.
Does pre-training a 3D transformer on quantum-mechanical properties help ADMET prediction?
QuADMET-Former pre-trains a 1.3M-parameter TorchMD-ET equivariant transformer on PubChemQC quantum properties — total energy, HOMO–LUMO gap, dipole moment, E3FP fingerprints and Löwdin charges — across ten released checkpoints, then adapts it by linear probing followed by fine-tuning. Both a fine-tuned arm and its frozen embedding read by TabPFN 3 are run on the authors' code and weights, against LightGBM, fine-tuned CheMeleon, Monroe + TabPFN 3 and fine-tuned Atom-JEPA.
What came out. Pre-trained LP-FT beats from-scratch training on 14 of 15 endpoints by a median +0.083 R², and LightGBM on 38 of 45 combinations. Against the pre-trained field it reaches the top Tukey tier on 5 of 27 ExpansionRx combinations, nearly splits with fine-tuned CheMeleon (15 wins, 9 ties, 21 losses over both data sets), and trails fine-tuned Atom-JEPA (2 wins, 35 losses) and Monroe + TabPFN 3 (10 wins, 33 losses).
Puts to the test
Six discrepancies between the paper and the authors' repository were found and are documented in the study README; the arms here use the released checkpoints with those issues worked around.
6 methods · 15 endpoints · 10 released QM checkpoints · 1.2 h GPU time
Code: studies/quadmet-former · Report builder: 08_build_page.py
Does a 3D equivariant encoder pre-trained on 19 million conformers beat the best frozen encoder?
Atom-JEPA pre-trains a 39M-parameter EquiformerV3 by latent prediction on 19M Uni-Mol conformers and fine-tunes it, multitask, on ten conformers per molecule. It is run here with the authors' own code and released weights, against the arm the paper did not run, Monroe + TabPFN 3, and a frozen Atom-JEPA arm read by the same TabPFN 3 call separates the representation from the fine-tuning.
What came out. Fine-tuned Atom-JEPA is ahead of fine-tuned CheMeleon on 38 of 45 combinations and of LightGBM on all 45. Against Monroe it is ahead on 6 of 9 ExpansionRx endpoints and behind on 4 of 6 Biogen ones, and training-set size does not explain the split (ρ = −0.04). Frozen, its embedding never beats Monroe's, a median −0.09 R²; fine-tuning never loses to freezing and adds +0.13 on ExpansionRx and +0.07 on Biogen.
Puts to the test
7 arms · 15 endpoints · 50 fine-tuned fold models · 46 h GPU time
Code: studies/atom-jepa · Report builder: 08_build_page.py
Does correcting each prediction with the known errors of similar training molecules help?
MolPAIR corrects one tabular foundation model with another: a global TabPFN 3 reads a frozen CheMeleon fingerprint, and a second TabPFN 3 learns from pairs of training molecules how the first model's error changes between similar molecules, so each test molecule borrows the errors of up to 32 neighbours. It is put beside LightGBM, fine-tuned CheMeleon and Monroe + TabPFN 3.
What came out. Against the prediction it corrects, MolPAIR is significantly ahead on 41 of 45 endpoint × metric combinations and behind on 2, and improves 51–66% of individual predictions, close to the paper's 55–57%. The gain holds, at half the size, on the test molecules the paper's own split would have kept. But the correction is worth +0.024 R², fine-tuning CheMeleon +0.047 and Monroe's embedding +0.129: MolPAIR loses to fine-tuned CheMeleon on 30 combinations and to Monroe + TabPFN 3 on 34.
Puts to the test
A reimplementation. The paper's code link does not resolve, so this is written from its Section 3 and Appendix A.
6 arms · 15 endpoints · 375 fold models · 41/45 ahead of its baseline
Code: studies/molpair · Report builder: 06_build_page.py
Holding Monroe's frozen embedding fixed, which in-context tabular model reads it best?
Every tabular foundation model in NVIDIA's structured-data-models package — TabICLv2, Kumo Tabular small, medium and large, and the 1.6-billion-parameter TabFM — reads the same frozen 720-d Monroe embedding on the same folds as Monroe + TabPFN 3, so the only thing that differs between arms is the head. The authors' own tabicl package, loading the same checkpoint as NVIDIA's TabICLv2 port, is the control.
What came out. Kumo Tabular large is on top of all 45 endpoint × metric combinations after Tukey's correction, the only arm never significantly worse than the best. In paired tests it is ahead of TabPFN 3 on 28 and behind on 6, and ahead of TabFM on 28 against 7 for an eighth of the GPU time. NVIDIA's TabICLv2 port reproduces upstream tabicl to within 0.008 R². Swapping heads moves R² by a median of 0.042 per endpoint, against 0.208 for the representation.
Puts to the test
9 arms · 5 heads fitted · 15 endpoints · 45/45 Kumo large on top
Code: studies/sdm-tabular · Report builder: 05_build_page.py
A paper says nearest-neighbour lookup matches modern ML under every split, and that all of them collapse to a naive baseline out of distribution. Does that hold against models it did not have?
Eleven arms on the ten ChEMBL potency sets and six splitting schemes of Lee, Moldagulov and Grzybowski, run on that paper's own folds: four methods fitted here, the paper's naive baseline, and the paper's own six arms imported at their published per-fold numbers. 1,500 fold models, and a re-implementation of the paper's k-NN that reproduces 250 of 250 of its released folds to 4e-9, which is what licenses putting the two sources on one axis.
What came out. Half the paper replicates. Error does climb as the splits harden, but so does the naive baseline it is measured against, and reading the gap instead of the raw MAE splits the two claims apart. k-NN is at the top of none of the 18 scheme x metric combinations and Monroe's frozen encoder is at the top of all 18, separating from k-NN on all six schemes and on five of six under a conservative target-level test. On the hardest schemes k-NN takes 0.005 and 0.010 pIC50 off the naive predictor and the paper's own GNN goes negative, while Monroe holds 0.121 and 0.141. The decay rates are nearly identical across arms, 0.81 to 0.98; what differs is the offset, so nothing here is more robust to distance, it is just uniformly better.
Puts to the test
The paper's six arms are its own published per-fold metrics, not re-run, and each had 50 Optuna trials per fold where the four arms fitted here use library defaults. The upstream release ships two disagreeing copies of the splits; the .pkl copy is used, because it reproduces the paper's numbers and the .json copy does not.
11 arms · 6 splitting schemes · 1,500 fold models fitted · 250/250 published folds reproduced · 0/18 combinations k-NN leads
Code: studies/split-fragility · Report builder: 09_build_page.py
Which foundation model, and does it replicate?
Nine modelling approaches over fifteen ADME and physicochemical endpoints on two unrelated data sets, 25 replicate models each, every one scored on a held-out test set it never saw. Four of the nine are pre-trained molecular foundation models, and only one of those four wins anything at all. The one that wins does no downstream training whatsoever — it freezes a pre-trained encoder and predicts each endpoint in context — and it appears three times, under three different tabular models, which is what lets the page price the representation against the predictor bolted onto it.
What came out. Monroe's frozen encoder is at the top of 21 of 27 ExpansionRx combinations and all 18 Biogen ones; MEGA-CL and Mol-JEPA are best on nothing, anywhere. Crossing two frozen representations with two in-context heads settles which half does the work: changing the head moves R² by a median of 0.016 and not even in a consistent direction, while changing the representation moves it by 0.168 on fifteen endpoints out of fifteen. The representation is worth about ten times the head.
Puts to the test
9 methods · 15 endpoints · 2,925 fold models · 3.19M predictions kept
Code: studies/expansion-ml-comparison · Report builder: 06_build_page.py
A graph network taught to predict fingerprints, against the fingerprints
PT-GIN is a Graph Isomorphism Network pre-trained on 462,189 QMugs molecules to predict each molecule's own 2,048-bit hashed ECFP4, then frozen and handed to LightGBM. Its authors report it beating hashed ECFP on five of six Biogen ADME tasks. Here it is run against those same fingerprints through that same predictor, so the only thing that differs between the two arms is the representation.
What came out. It wins six endpoints of fifteen and loses six; on Biogen, its own benchmark, it wins one and loses three. Which six it wins is not about how much data an endpoint has — it is about how well the fingerprints were already doing. PT-GIN helps where the baseline was weak and hurts where it was strong, Spearman ρ = −0.65 with the baseline R².
Puts to the test
5 methods · 15 endpoints · 1,425 fold models · 10 released checkpoints
Code: studies/ecfp-pretrain · Report builder: 06_build_page.py
Unimodal versus multimodal, on ADME data
Thirty-three ways of combining four molecular representations — RDKit descriptors, Mol2Vec, a supervised GNN's graph embedding and a character BiGRU's SMILES embedding — crossed with early and late fusion and three final learners, over the same fifteen endpoints. 12,375 fitted configurations and 6,375 more for the two controls.
What came out. The paper's central finding replicates: fusing modalities moves accuracy very little, and none of it survives a correction for multiple comparisons. What fusion buys is calibration. Two things the paper could not see from inside its own design also show up — a single well-initialised graph network beats all thirty-three configurations on both data sets, and when the GNN modality is actually a learned representation, which in the released code it is not, the ranking of which modality matters inverts.
Puts to the test
A reimplementation. The authors' MIT-licensed release is a set of Colab dumps that cannot be run, so this is written from that source read as a specification alongside the paper's Supporting Information.
33 configurations · 15 endpoints · 18,750 fits · 4 modalities
Code: studies/multimodal-fusion · Report builder: 12_build_page.py
Is picking a model per endpoint worth it?
Trimole-Hybrid is not a model but a selection procedure. For every endpoint and every fold it builds sixty candidate predictors — five molecular views crossed with three blocks of chemistry priors crossed with four backends — fits all of them, scores them on the held-out validation fifth, and keeps exactly one to predict the test set. It is put against four fixed architectures on the same fifteen endpoints.
What came out. It loses to a single well-initialised graph network on both data sets, for two reasons only visible in what it selected: nothing in its pool learns a representation, and on ExpansionRx the split it selects on does not resemble the split it is scored on. What it chose does not replicate either.
Puts to the test
A reimplementation. The authors' release describes itself as not runnable, so this reproduces the method, not the paper's numbers.
5 methods · 60 candidates per fold · 22k candidate fits · 1,425 fold models
Code: studies/expansion-ml-comparison · Report builder: 14_build_trimole_page.py
studies/<slug>/ one self-contained study: scripts, README, requirements,
per-fold metrics, tables, figures, and a study.json
site/build_site.py reads every study.json, writes docs/
site/overview.py the cross-study Tukey comparison at the top of the index
site/style.py the stylesheet the index shares with the reports
docs/ the GitHub Pages site
docs/index.html the overall report, one card per study report
docs/reports/<slug>.html each study's own report, copied from its results/
docs/overview.json the tally table, and docs/assets/ the figures with it
A study directory stands on its own. It carries its own config.py,
requirements.txt, README and results, and it is run from inside itself, so it works
exactly as it did before it was collected here and can be lifted back out.
The index needs nothing but a Python interpreter:
python site/build_site.py # rebuild docs/ and the card block above
python site/build_site.py --check # say what is stale, change nothing
That copies each study's built report into docs/reports/ and regenerates the index
and the card block in this README from the study.json files. It does not run any
science. A study's report is built inside that study's own directory, in the
environment that study needs — see its README for the commands and its
requirements.txt for the environment.
The cross-study comparison at the top of the index is the one part that does read the numbers, so it is a separate script and needs pandas, statsmodels and matplotlib:
python site/overview.py # -> docs/overview.json, docs/assets/*.png
python site/build_site.py # embeds what it wrote
The table and the figures use the same metric, so a row and its panel can never
disagree about who won: one Tukey HSD per endpoint, counted best alone / tied /
worse. That metric is MAE by default, since it is in the units of the endpoint;
--metric r2 or --metric spearman rebuilds both against either of the others.
Run it when a study's fold_metrics.csv changes. It refuses to pool anything
unless the four reference arms are identical across the studies that carry them,
and it asserts the tie rule on every endpoint × metric combination before
counting.
Drop the study directory under studies/<slug>/. It keeps its own README,
requirements.txt and results.
Add a studies/<slug>/study.json:
{
"slug": "my-study",
"name": "my-study",
"datasets": ["ExpansionRx", "Biogen ADME"],
"endpoints": 15,
"protocol": "5 x 5 GroupKFold over chemical clusters, one fixed holdout, Tukey HSD",
"updated": "2026-09-02",
"reports": [
{
"slug": "my-study",
"date": "2026-09-02",
"title": "The title on the card and the report",
"eyebrow": "What kind of question this is",
"question": "The one-line question the study answers",
"summary": "Two or three sentences on what was compared and how much of it.",
"finding": "What came out, in one or two sentences, including the number that matters.",
"facts": [{"n": "15", "label": "endpoints"}],
"validates": [
{
"name": "Short name for the method",
"authors": "Author, A.; Author, B.; et al.",
"title": "The paper's full title",
"venue": "J. Chem. Inf. Model. 2026",
"label": "doi:10.1021/...",
"url": "https://doi.org/..."
}
],
"caveat": "Optional. Say so here if this is a reimplementation.",
"source": "results/report.html",
"builder": "06_build_page.py"
}
]
}
slug, title, question, summary, finding and source are required;
everything else is optional. A study with two reports — as
expansion-ml-comparison has — lists both, and gets a card for each.
Cards are ordered newest first, by each report's date, so a new study lands
at the top on its own and nothing has to be renumbered. A report without a
date falls back to its study's updated, and one with neither sorts last.
Reports built on the same day are separated by their study's slug, which only
keeps the order stable and means nothing.
Each entry in validates is a paper the study puts to the test, cited in
full on the card: title, authors and venue are printed, label is the
link text and url is where it points.
Run python site/build_site.py.
If the new study reuses the reference arms, point its importer at
studies/expansion-ml-comparison; studies/ecfp-pretrain/00_import_baselines.py is
the worked example.
Code is MIT — see LICENSE. Two data files travel with the studies that use them and keep their own terms:
expansion_log_scaled.csv — the OpenADMET–ExpansionRx blind challenge data,
contributed by Expansion Therapeutics, released by OpenADMET under CC BY 4.0
(doi:10.57967/hf/9687), and put on a
log10(x + 1) scale here. Please credit both.biogen_adme_3521.csv — built from
molecularinformatics/Computational-ADME,
released by Biogen under the MIT License with the paper of Fang et al.No third-party pre-trained weights are vendored. Each study's README says where its checkpoints come from and under what licence.