EXSCLAIM2.0: LLM-powered Automatic EXtraction, Separation, and Caption-based natural Language Annotation of IMages from scientific figures

🤔 Consider Collaboration

If you find this tool or any of its derived capabilities useful, please consider registering as a user of Center for Nanoscale Materials. We will keep you posted of latest developments, as well as opportunities for computational resources, relevant data, and collaboration. Please contact Maria Chan (mchan@anl.gov) for details.

Introduction to EXSCLAIM2.0

EXSCLAIM2.0 is a Python package combining EXSCLAIM! code with Large Language models (LLMs) that can be used for the automatic generation of datasets of labelled images from published papers. It in three main steps:

1. [JournalScraper]: scrap journal websites, acquiring figures, captions, and metadata
2. HTMLScraper: scrap user provided HTML files, acquiring figures, captions, and metadata
3. CaptionDistributor: separate figure captions into the component chunks that refer to the figure's subfigures using LLMs and prompt engineering
4. FigureSeparator: separate figures into subfigures, detect scale information, label, and type of image

Examples and tutorials

We provide several tutotials demonstrating how to use EXSCLAIM2.0:

1. Nature_exsclaim_search: automatically scrapping data from literature and performing Named Entity Recognition (NER) on the extracted captions.
2. HTMLScrapper: automatically scrapping data from user provided HTML files
3. Microscopy_CLIP_retrieval: Using Microscopy_CLIP to perform image-to-image and text-to-image retrieval on our multimodal microscopy dataset.

Using EXSCLAIM

Requirements

EXSCLAIM works with Python 3.6+. We recommend using a conda or python environment to install dependencies. To use the pipeline, you need a Query on which to run the pipeline. The query can be a JSON or Python dictionary (depending on how you are accessing the pipeline) and must have the parameters(/keys/attributes) defined in the Query JSON schema and examples can be found in the query directory.

Installation with Git

To install directly from github, run the following commands (it is recommended to run in a conda or python virtual environment):

git clone https://github.com/MaterialEyes/exsclaim.git
cd exsclaim
pip setup.py install
python load_models.py

Scrapping data from dynamic journal webpages

If scrapping data from Journals that use javascript (e.g. RSC, ACS) you need to setup chome-driver and chome and add their path to the exsclaim/journal.py file.

e.g. for a linux terminal you need the following:
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
wget https://chromedriver.storage.googleapis.com/90.0.4430.24/chromedriver_linux64.zip

If you run into errors, please check Troubleshooting. If they persist, please open an issue.

Acknowledgements

This material is based upon work supported by Laboratory Directed Research and Development (LDRD) funding from Argonne National Laboratory, provided by the Director, Office of Science, of the U.S. Department of Energy under Contract No. DE-AC02-06CH11357

This work was performed at the Center for Nanoscale Materials, a U.S. Department of Energy Office of Science User Facility, and supported by the U.S. Department of Energy, Office of Science, under Contract No. DE-AC02-06CH11357.

We gratefully acknowledge the computing resources provided on Bebop, a high-performance computing cluster operated by the Laboratory Computing Resource Center at Argonne National Laboratory.

Citation

If you find EXSCLAIM2.0 useful, please encourage its development by citing the following papers in your research:

- Aikaterini Vriza, Eric Schwenker, Carter Ekberg, Coltin Kunz, Harris Heit, Maria K Y Chan, 2024
10.1093/mam/ozae044.166.

- Schwenker, E., Jiang, W. Spreadbury, T., Ferrier N., Cossairt, O., Chan M.K.Y., EXSCLAIM! - An automated pipeline for the construction and
labeling of materials imaging datasets from scientific literature. arXiv e-prints (2021): arXiv-2103

=======

exsclaim2.0