SMILES eval

This is a dataset that measures LLM capabilities at generating SMILES chemical molecule representations from natural language descriptions. It was generated by prompting Gemini 2.5 Pro for molecule description and SMILES pairs, and filtering for a) molecules valid according to rdkit and b) reliably regenerated by Gemini and c) removing duplicates using fuzzy matching. Difficulty is estimated by how reliable Gemini 2.5 Pro is at generating the molecule.

This dataset was used for evaluation in the paper Constrained Decoding of Diffusion LLMs with Context-Free Grammars. You can find the corresponding evaluation code on the project GitHub Repository.

Example Usage

from datasets import load_dataset
import json

dataset = load_dataset('eth-sri/smiles-eval')
for instance in dataset['test']:
    print(json.dumps(instance, indent=2))
    break

Example Instance

{
  "instance_id": "smiles_0",
  "input": "Pyridine, a six-membered aromatic heterocyclic organic compound containing one nitrogen atom, replacing a C-H unit in a benzene ring.",
  "output": "n1ccccc1",
  "difficulty_category": "medium"
}