Description

This dataset is a geoscience-specific subset of CommonCrawl used for GeoGPT training. CommonCrawl is a free and open repository of web crawl data with over 250 billion web pages and is widely used by leading large language models. We apply data mining algorithms to extract geoscience-related content from this vast dataset.

This dataset comprises 12,414,268 samples, each containing the following metadata to trace the data source within CommonCrawl:

• id (string): The original unique identifier for each sample from CommonCrawl.
• dump (string): The specific CommonCrawl dump in which each sample was retrieved.

Note

This dataset is primarily intended to support geoscience research by serving as a training corpus for large language models. It is specifically designed for non-commercial research and educational purposes.

The dataset is not intended for use in any manner that violates applicable laws or regulations, nor for any activities prohibited by the license agreement.