LLM Classification Pipeline

A modular pipeline for text classification using large language models, featuring label extraction, clustering, hierarchical taxonomy mapping, and sentiment analysis.

This implementation is based on the methodology described in:

Wei, M., Zhang, P., Chen, C. et al. Mapping AI ethics narratives: evidence from Twitter discourse between 2015 and 2022. Humanit Soc Sci Commun 12, 212 (2025). https://doi.org/10.1057/s41599-025-04469-9

Overview

This pipeline automates the process of extracting, organizing, and analyzing thematic structures from large-scale text corpora. It combines multiple state-of-the-art NLP models to:

1. Extract keywords/labels from unstructured text
2. Cluster and align semantically similar labels
3. Map labels to hierarchical taxonomies for organized analysis
4. Calculate statistics across the taxonomy hierarchy
5. Analyze sentiment to understand emotional tone

This pipeline integrates multiple pre-trained transformer models:

1. KeyBART (Label Extraction)

• Model: bloomberg/KeyBART
• Purpose: Extract keywords and key phrases from text
• Type: Sequence-to-sequence model fine-tuned for keyphrase extraction
• Output: Semicolon-separated list of keywords/labels

2. Sentence Transformers (Label Clustering)

• Model: all-MiniLM-L6-v2
• Purpose: Generate semantic embeddings for label similarity
• Type: Sentence embedding model
• Output: 384-dimensional dense vectors for clustering

3. RoBERTa Large MNLI (Taxonomy Mapping)

• Model: roberta-large-mnli
• Purpose: Zero-shot classification to map labels to taxonomy categories
• Type: Natural language inference model
• Output: Probability scores for category membership

4. Twitter RoBERTa Sentiment (Sentiment Analysis)

• Model: cardiffnlp/twitter-roberta-base-sentiment-latest
• Purpose: Analyze sentiment polarity of text
• Type: Sequence classification model fine-tuned on Twitter data
• Output: Positive/Negative/Neutral sentiment scores

5. DBSCAN (Label Clustering)

• Algorithm: Density-Based Spatial Clustering of Applications with Noise
• Purpose: Group semantically similar labels together
• Output: Clustered label groups with noise filtering

OpenAI GPT Models (Optional)

For enhanced performance, you can optionally use OpenAI's GPT models:

GPT-4o / GPT-4o-mini (Label Extraction & Taxonomy)

• Models: gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo
• Purpose: Extract labels, generate taxonomy, map labels to categories
• Advantages: Better understanding of context, more nuanced extraction
• Requirements: OpenAI API key, incurs API costs
• Use cases:
  • Label extraction with better semantic understanding
  • Automatic taxonomy structure generation
  • Context-aware label-to-category mapping

Features

• Automated Label Extraction: Extract keywords/labels using KeyBART transformer model
• Semantic Clustering: Clean, lemmatize and cluster similar labels using sentence embeddings and DBSCAN
• Hierarchical Taxonomy Mapping: Map labels to custom taxonomies using zero-shot classification
• Statistical Analysis: Calculate frequency distributions across hierarchical categories
• Sentiment Analysis: Analyze emotional tone using Twitter-optimized RoBERTa model
• Parallel Processing: Multi-process support for efficient large-scale processing
• Flexible Input: Support for JSONL and Excel file formats

Project Structure

llm_classification/
├── src/
│   ├── core/
│   │   ├── clustering.py      # Label clustering and alignment
│   │   ├── pipeline.py        # Main pipeline orchestrator
│   │   ├── statistics.py      # Statistics calculation
│   │   └── taxonomy.py        # Taxonomy mapping
│   ├── models/
│   │   ├── label_extractor.py # KeyBART label extraction
│   │   └── sentiment_analyzer.py # RoBERTa sentiment analysis
│   └── utils/
│       ├── data_io.py         # Data I/O utilities
│       ├── device.py          # Device management
│       └── text_processing.py # Text preprocessing
├── config/                     # Configuration files
├── tem_file/                   # Cache directory
├── main.py                     # Main CLI entry point
├── extract_labels.py          # Label extraction script
├── analyze_sentiment.py       # Sentiment analysis script
└── cluster_labels.py          # Label clustering script

Installation

Requirements

• Python 3.8+
• CUDA-compatible GPU (optional, but recommended for faster processing)

Install Dependencies

pip install -r requirements.txt

Or manually install:

pip install torch transformers sentence-transformers scikit-learn scipy pandas nltk tqdm openpyxl

Download Models

All models download automatically on first use! No manual download needed.

The pipeline uses models from Hugging Face Hub (~3.6 GB total):

• KeyBART (~1.2 GB) - Label extraction
• all-MiniLM-L6-v2 (~90 MB) - Sentence embeddings
• roberta-large-mnli (~1.4 GB) - Taxonomy mapping
• twitter-roberta-base-sentiment (~500 MB) - Sentiment analysis
• bert-base-uncased (~440 MB) - Text processing

First run: Models download automatically (5-30 minutes) Subsequent runs: Uses cached models (instant, works offline)

Pre-download all models (optional):

python download_models.py

For China users (if Hugging Face is slow):

export HF_ENDPOINT=https://hf-mirror.com
python main.py --input data.jsonl

Model cache location: ~/.cache/huggingface/

Download NLTK Data

The pipeline will automatically download required NLTK resources on first run:

• WordNet lexical database
• Penn Treebank POS tagger
• Open Multilingual Wordnet

Optional: Install OpenAI for GPT Support

To use GPT-based features:

pip install openai>=1.12.0

Set your API key:

export OPENAI_API_KEY='your-api-key-here'

Usage

Standard Pipeline (Open-Source Models)

Use free, open-source models (recommended for most cases):

python main.py \
    --input data.jsonl \
    --column content \
    --taxonomy taxonomy.json \
    --processes 4 \
    --sentiment

GPT Pipeline (Optional)

Use OpenAI GPT models for enhanced performance:

# Set API key first
export OPENAI_API_KEY='your-api-key-here'

# Run GPT pipeline
python main_gpt.py \
    --input data.jsonl \
    --column content \
    --model gpt-4o-mini \
    --generate-taxonomy \
    --batch-size 50

GPT Pipeline Options:

• --model: Choose GPT model (gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo)
• --generate-taxonomy: Auto-generate taxonomy from labels
• --taxonomy: Use existing taxonomy file
• --api-key: Pass API key directly (or use env var)
• --batch-size: Labels per GPT request (default: 50)
• --max-labels: Max labels for taxonomy generation (default: 500)

Individual Steps

Extract Labels:

python extract_labels.py \
    --file_path data.jsonl \
    --col_name content \
    --processes_num 3

Cluster Labels:

python cluster_labels.py \
    --file_path data_labels.jsonl \
    --min_samples 3

Analyze Sentiment:

python analyze_sentiment.py \
    --file_path data.jsonl \
    --col_name content \
    --processes_num 5

Parameters

• --input: Input file (.jsonl or .xlsx)
• --column: Text content column name (default: content)
• --taxonomy: Path to taxonomy JSON file
• --processes: Number of parallel processes (default: 3)
• --min-samples: Minimum samples for DBSCAN (default: 3)
• --max-length: Max output length for labels (default: 100)
• --num-beams: Number of beams for generation (default: 10)
• --threshold: Taxonomy mapping threshold (default: 0.9)
• --sentiment: Include sentiment analysis
• --skip-extraction: Skip label extraction step
• --overwrite: Overwrite cached results

Input Format

JSONL format with one JSON object per line:

{"content": "Your text here", "other_field": "value"}

Output Files

• {input}_labels.jsonl: Extracted labels
• {input}_sentiment.jsonl: Sentiment scores
• {input}_hierarchy_labels.jsonl: Mapped hierarchical labels
• tem_file/sorted_word_list_*.json: Sorted unique labels
• tem_file/final_mapping_dict*.json: Label alignment mapping
• tem_file/all_num_key*.json: Category frequency statistics

Taxonomy Format

The taxonomy should be a nested JSON structure representing hierarchical categories:

{
  "Category1": {
    "Subcategory1": ["label1", "label2"],
    "Subcategory2": ["label3", "label4"]
  },
  "Category2": {
    "Subcategory3": ["label5", "label6"]
  }
}

Example taxonomy from the AI Ethics research paper:

{
  "Technology": {
    "AI Systems": ["artificial intelligence", "machine learning", "algorithm"],
    "Data": ["data privacy", "personal data", "data protection"]
  },
  "Ethics": {
    "Bias and Fairness": ["bias", "discrimination", "fairness"],
    "Transparency": ["explainability", "transparency", "accountability"]
  },
  "Society": {
    "Impact": ["social impact", "employment", "inequality"],
    "Regulation": ["regulation", "policy", "governance"]
  }
}

Pipeline Workflow

Input Text (JSONL/Excel)
    ↓
[1] Label Extraction (KeyBART)
    → Extracts keywords/labels from each document
    ↓
[2] Label Cleaning & Clustering (NLTK + Sentence-BERT + DBSCAN)
    → Lemmatizes words
    → Generates semantic embeddings
    → Clusters similar labels
    → Aligns label variants
    ↓
[3] Taxonomy Mapping (RoBERTa-MNLI)
    → Maps labels to hierarchical categories
    → Uses zero-shot classification
    ↓
[4] Statistics Calculation
    → Counts label frequencies
    → Aggregates parent category counts
    ↓
[5] Sentiment Analysis (Twitter-RoBERTa) [Optional]
    → Analyzes sentiment polarity
    ↓
Output: Labeled & Classified Documents + Statistics

GPT Pipeline Workflow (Optional)

Input Text (JSONL/Excel)
    ↓
[1] Label Extraction (GPT-4o/GPT-4o-mini)
    → Uses GPT to extract contextual labels
    ↓
[2] Label Cleaning & Clustering (NLTK + Sentence-BERT + DBSCAN)
    → Same as standard pipeline
    ↓
[3A] Taxonomy Generation (GPT-4o) [Optional]
    → Auto-generate hierarchical taxonomy
    OR
[3B] Taxonomy Mapping (GPT-4o-mini)
    → Map labels to provided taxonomy
    ↓
[4] Statistics Calculation
    → Same as standard pipeline
    ↓
Output: Labeled & Classified Documents + Statistics

Citation

If you use this pipeline in your research, please cite:

@article{wei2025mapping,
  title={Mapping AI ethics narratives: evidence from Twitter discourse between 2015 and 2022},
  author={Wei, M. and Zhang, P. and Chen, C. and others},
  journal={Humanities and Social Sciences Communications},
  volume={12},
  pages={212},
  year={2025},
  publisher={Palgrave Macmillan},
  doi={10.1057/s41599-025-04469-9}
}