MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation

This is the official dataset repository for the ICLR 2026 paper: MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation.

The code for using and evaluating the MolLangBench datasets is provided in this GitHub repository.

MolLangBench is a comprehensive benchmark designed to evaluate the fundamental capabilities of AI models in language-prompted molecular structure recognition, editing, and generation.

---

Dataset Structure

MolLangBench consists of three core tasks:

• Molecular Structure Recognition:
  Evaluate a model's ability to interpret and reason over molecule representations (SMILES strings, images) from natural language prompts.
• Molecule Editing:
  Assess a model's capability to apply language-based structural edits to molecules.
• Molecule Generation:
  Evaluate a model's ability to generate new molecule structures based on text prompts.

For complete code, usage instructions, and evaluation pipelines, please visit our GitHub repository.

---

Benchmark Results

Molecular Structure Recognition

Click to view the full results table

Task	GPT-4o	GPT-4.5	GPT-4.1	o1-mini	o1	o3-mini	o3	o4-mini	GPT-5	Gemini-2.5-Pro	Claude-Opus-4.1	DeepSeek-R1	R1-70B	Llama-4	Qwen3-Max	o3 (image)	o4-mini (image)
One-hop neighbors	0.355/0.140	0.600/0.425	0.570/0.330	0.735/0.640	0.825/0.720	0.870/0.820	0.935/0.895	0.880/0.845	0.950/0.920	0.905/0.840	0.860/0.805	0.825/0.710	0.585/0.430	0.520/0.290	0.350/0.130	0.890/0.855	0.840/0.780
Two-hop neighbors	0.215/0.055	0.280/0.100	0.400/0.210	0.465/0.350	0.745/0.560	0.820/0.740	0.935/0.825	0.870/0.790	0.940/0.900	0.885/0.775	0.790/0.670	0.610/0.475	0.305/0.135	0.245/0.065	0.245/0.030	0.770/0.705	0.775/0.690
Three-hop neighbors	0.165/0.015	0.355/0.165	0.265/0.140	0.400/0.265	0.560/0.400	0.825/0.705	0.925/0.830	0.775/0.710	0.925/0.895	0.820/0.665	0.660/0.525	0.550/0.385	0.300/0.130	0.210/0.055	0.100/0.025	0.695/0.600	0.660/0.575
Quaternary carbons	0.530/0.290	0.690/0.435	0.740/0.440	0.615/0.470	0.865/0.665	0.835/0.740	0.935/0.865	0.845/0.750	0.980/0.910	0.945/0.815	0.875/0.720	0.440/0.330	0.780/0.680	0.345/0.205	0.170/0.070	0.670/0.600	0.720/0.665
Ring junctions	0.285/0.080	0.495/0.185	0.485/0.210	0.325/0.175	0.575/0.470	0.580/0.520	0.685/0.650	0.590/0.570	0.830/0.810	0.860/0.740	0.655/0.530	0.535/0.420	0.255/0.160	0.385/0.180	0.210/0.650	0.660/0.595	0.615/0.555
Bond connection	0.448	0.472	0.336	0.698	0.758	0.832	0.950	0.880	0.935	0.855	0.860	0.802	0.564	0.590	0.530	0.626	0.706
Halogen atoms	0.845/0.290	0.905/0.420	0.900/0.355	0.920/0.570	0.975/0.740	0.955/0.710	0.965/0.860	0.965/0.820	1.000/0.925	1.000/0.820	0.990/0.860	0.970/0.735	0.740/0.375	0.865/0.455	0.830/0.265	0.855/0.815	0.920/0.860
Aldehyde	0.855/0.570	0.965/0.610	0.945/0.730	0.855/0.725	0.970/0.825	0.985/0.920	0.990/0.960	0.985/0.945	1.000/0.965	1.000/0.920	0.995/0.950	0.960/0.835	0.715/0.585	0.900/0.645	0.700/0.510	0.925/0.925	0.975/0.965
Amide	0.505/0.180	0.570/0.205	0.635/0.315	0.585/0.340	0.715/0.440	0.685/0.510	0.765/0.650	0.755/0.610	0.900/0.775	0.920/0.640	0.715/0.585	0.635/0.415	0.495/0.205	0.520/0.195	0.450/0.120	0.565/0.500	0.735/0.665
Carboxyl	0.760/0.260	0.885/0.235	0.900/0.485	0.840/0.580	0.965/0.675	0.955/0.760	0.985/0.845	0.950/0.725	0.980/0.875	0.990/0.845	0.960/0.715	0.900/0.660	0.820/0.495	0.875/0.435	0.710/0.235	0.785/0.750	0.870/0.820
Ester	0.600/0.145	0.760/0.285	0.780/0.330	0.675/0.325	0.935/0.500	0.895/0.645	0.955/0.780	0.950/0.640	0.985/0.800	0.975/0.705	0.915/0.590	0.680/0.400	0.615/0.270	0.710/0.220	0.500/0.130	0.720/0.505	0.840/0.595
Ketone	0.530/0.155	0.750/0.260	0.870/0.435	0.750/0.465	0.925/0.600	0.985/0.745	0.985/0.865	0.985/0.795	1.000/0.885	0.985/0.825	0.955/0.725	0.880/0.600	0.770/0.370	0.815/0.370	0.575/0.200	0.765/0.675	0.850/0.775
Benzene	0.490/0.145	0.540/0.105	0.660/0.155	0.530/0.235	0.720/0.360	0.725/0.565	0.880/0.695	0.730/0.550	0.925/0.840	0.715/0.470	0.695/0.500	0.595/0.385	0.500/0.190	0.590/0.195	0.455/0.105	0.675/0.405	0.680/0.485
Furan	0.295/0.265	0.820/0.325	0.905/0.515	0.780/0.500	0.920/0.660	0.865/0.745	0.975/0.845	0.940/0.790	0.995/0.915	0.905/0.740	0.940/0.785	0.895/0.710	0.850/0.490	0.935/0.445	0.715/0.325	0.890/0.820	0.870/0.815
Pyridine	0.555/0.225	0.525/0.250	0.730/0.365	0.685/0.375	0.765/0.555	0.860/0.740	0.925/0.825	0.835/0.750	0.915/0.865	0.815/0.720	0.770/0.620	0.685/0.520	0.630/0.340	0.675/0.270	0.485/0.190	0.715/0.585	0.790/0.665
Thiophene	0.860/0.385	0.840/0.325	0.880/0.480	0.840/0.605	0.915/0.690	0.940/0.795	0.970/0.890	0.925/0.820	1.000/0.945	0.925/0.775	0.975/0.820	0.920/0.705	0.850/0.565	0.930/0.455	0.785/0.350	0.960/0.855	0.920/0.855
Bond stereo	0.390	0.395	0.670	0.425	0.330	0.310	0.480	0.325	0.655	0.295	0.530	0.310	0.345	0.520	0.470	0.575	0.640
Chiral stereo	0.440	0.395	0.530	0.465	0.510	0.435	0.545	0.520	0.700	0.545	0.510	0.440	0.495	0.420	0.465	0.510	0.495
Average	0.507/0.249	0.625/0.311	0.678/0.391	0.644/0.456	0.776/0.581	0.798/0.680	0.877/0.792	0.817/0.713	0.923/0.862	0.852/0.753	0.814/0.692	0.721/0.566	0.571/0.360	0.614/0.295	0.486/0.186	0.736/0.661	0.772/0.700

• Each entry reports recognition accuracy / localization accuracy where applicable.
• Tasks with only recognition evaluation show a single recognition accuracy value.
• Bold values indicate the best performance among all evaluated language models.
• "o3 (image)" and "o4-mini (image)" indicate vision-language models evaluated on molecular images.

---

Molecule Editing and Generation Benchmark Results

Click to view editing and generation results

Task	GPT-4o	GPT-4.5	GPT-4.1	o1-mini	o1	o3-mini	o3	o3 (SELFIES)	o4-mini	GPT-5	DeepSeek-R1	R1-70B	Llama-4	Qwen3-Max	Gemini-2.5-Pro	Claude-Opus-4.1	GPT-Image-1
Molecule editing	0.725/0.591/0.400	0.950/0.823/0.570	0.835/0.693/0.465	0.710/0.589/0.385	0.845/0.788/0.635	0.805/0.758/0.650	0.945/0.903/0.785 (0.900/0.846/0.670)	0.960/0.474/0.195 (0.865/0.372/0.140)	0.920/0.860/0.690	0.945/0.918/0.855 (0.950/0.890/0.820)	0.720/0.643/0.485	0.675/0.565/0.375	0.895/0.772/0.545 (0.890/0.752/0.490)	0.690/0.561/0.360 (0.700/0.496/0.230)	0.930/0.881/0.745 (0.945/0.876/0.695)	0.950/0.884/0.705 (0.965/0.879/0.665)	0.135
Molecule generation	0.525/0.174/0.005	0.800/0.411/0.055	0.710/0.344/0.035	0.335/0.170/0.035	0.385/0.257/0.100	0.450/0.349/0.175	0.670/0.546/0.290 (0.695/0.569/0.360)	0.185/0.005/0.000 (0.080/0.004/0.000)	0.600/0.458/0.260	0.690/0.596/0.430 (0.820/0.735/0.590)	0.400/0.209/0.045	0.205/0.077/0.010	0.875/0.511/0.115 (0.870/0.557/0.190)	0.465/0.104/0.000 (0.550/0.163/0.050)	0.865/0.737/0.430 (0.955/0.833/0.555)	0.920/0.725/0.330 (0.970/0.790/0.490)	0.000

• Each entry reports Valid SMILES rate / Tanimoto similarity / task accuracy, measuring chemical validity, structural fidelity, and task success respectively.
• Results are shown for the core set; values in parentheses denote performance on the extended set.
• For image-generation models, only the final task accuracy is reported.
• Bold entries indicate the best performance among all evaluated models.

---

Citation

Please cite our paper if you use MolLangBench in your research:

@inproceedings{MolLangBench,
  title={MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation},
  author={Feiyang Cai and Jiahui Bai and Tao Tang and Guijuan He and Joshua Luo and Tianyu Zhu and Srikanth Pilla and Gang Li and Ling Liu and Feng Luo},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
}

License

This dataset is distributed under the MIT License.

Links and Contact

• GitHub: TheLuoFengLab/MolLangBench
• Discord: Join our community
• Maintainer: Feiyang Cai