Shadowgraph
TrendingAbout
Sign in

ivan-bilan/The-NLP-Pandect

A comprehensive reference for all topics related to Natural Language Processing

Python

2,039

167 commits

updated Jun 1, 2026

See the code

README

The-NLP-Pandect

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find almost anything related to Natural Language Processing that is available online.

Note Quick legend on available resource types:

⭐ - open source project, usually a GitHub repository with its number of stars

📙 - resource you can read, usually a blog post or a paper

🗂️ - a collection of additional resources

🔱 - non-open source tool, framework or paid service

🎥️ - a resource you can watch

🎙️ - a resource you can listen to

Table of Contents

📇 Main Section🗃️ Sub-sections Sample
NLP ResourcesPaper Summaries, Conference Summaries, NLP Datasets
NLP PodcastsNLP-only Podcasts, Podcasts with many NLP Episodes
NLP Newsletters-
NLP Meetups-
NLP YouTube Channels-
NLP BenchmarksGeneral NLU, Question Answering, Multilingual
Research ResourcesResource on Transformer Models, Distillation and Pruning, Automated Summarization
Industry ResourcesBest Practices for NLP Systems, MLOps for NLP
Speech RecognitionGeneral Resources, Text to Speech, Speech to Text, Datasets
Topic ModelingBlogs, Frameworks, Repositories and Projects
Keyword ExtractionText Rank, Rake, Other Approaches
Responsible NLPNLP and ML Interpretability, Ethics, Bias, and Equality in NLP, Adversarial Attacks for NLP
NLP FrameworksGeneral Purpose, Data Augmentation, Machine Translation, Adversarial Attacks, Dialog Systems & Speech, Entity and String Matching, Non-English Frameworks, Text Annotation
Learning NLPCourses, Books, Tutorials
NLP Communities-
Other NLP TopicsTokenization, Data Augmentation, Named Entity Recognition, Error Correction, AutoML/AutoNLP, Text Generation

The-NLP-Resources

Note Section keywords: paper summaries, compendium, awesome list

Compendiums and awesome lists on the topic of NLP:

NLP Conferences, Paper Summaries and Paper Compendiums:

Papers and Paper Summaries
Conference Summaries

NLP Progress and NLP Tasks:

NLP Datasets:

Word and Sentence embeddings:

Notebooks, Scripts and Repositories

Non-English resources and Compendiums

Pre-trained NLP models

NLP History

General
2020 Year in Review

The-NLP-Podcasts

🔙 Back to the Table of Contents

NLP-only podcasts

Many NLP episodes

Some NLP episodes

The-NLP-Newsletter

The-NLP-Meetups

The-NLP-Youtube

The-NLP-Benchmarks

🔙 Back to the Table of Contents

General NLU

  • ⭐ GLUE - General Language Understanding Evaluation (GLUE) benchmark
  • ⭐ SuperGLUE - benchmark styled after GLUE with a new set of more difficult language understanding tasks
  • ⭐ decaNLP - The Natural Language Decathlon (decaNLP) for studying general NLP models
  • ⭐ dialoglue - DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue [GitHub, 287 stars]
  • ⭐ DynaBench - Dynabench is a research platform for dynamic data collection and benchmarking
  • ⭐ Big-Bench - collaborative benchmark for measuring and extrapolating the capabilities of language models [GitHub, 3244 stars]

Summarization

  • ⭐ WikiAsp - WikiAsp: Multi-document aspect-based summarization Dataset
  • ⭐ WikiLingua - A Multilingual Abstractive Summarization Dataset

Question Answering

  • ⭐ SQuAD - Stanford Question Answering Dataset (SQuAD)
  • ⭐ XQuad - XQuAD (Cross-lingual Question Answering Dataset) for cross-lingual question answering
  • ⭐ GrailQA - Strongly Generalizable Question Answering (GrailQA)
  • ⭐ CSQA - Complex Sequential Question Answering

Multilingual and Non-English Benchmarks

  • 📙 XTREME - Massively Multilingual Multi-task Benchmark
  • ⭐ GLUECoS - A benchmark for code-switched NLP
  • ⭐ IndicGLUE - Natural Language Understanding Benchmark for Indic Languages
  • ⭐ LinCE - Linguistic Code-Switching Evaluation Benchmark
  • ⭐ Russian SuperGlue - Russian SuperGlue Benchmark

Bio, Law, and other scientific domains

  • ⭐ BLURB - Biomedical Language Understanding and Reasoning Benchmark
  • ⭐ BLUE - Biomedical Language Understanding Evaluation benchmark
  • ⭐ LexGLUE - A Benchmark Dataset for Legal Language Understanding in English

Transformer Efficiency

Other

  • ⭐ CodeXGLUE - A benchmark dataset for code intelligence
  • ⭐ CrossNER - CrossNER: Evaluating Cross-Domain Named Entity Recognition
  • ⭐ MultiNLI - Multi-Genre Natural Language Inference corpus
  • ⭐ iSarcasm: A Dataset of Intended Sarcasm - iSarcasm is a dataset of tweets, each labelled as either sarcastic or non_sarcastic
  • ⭐ SLTev - tool for comprehensive evaluation of (simultaneous) spoken language translation [GitHub, 12 stars]

The-NLP-Research

🔙 Back to the Table of Contents

General

Embeddings

Repositories

Blogs

Cross-lingual Word and Sentence Embeddings

  • ⭐ vecmap - VecMap (cross-lingual word embedding mappings) [GitHub, 654 stars]
  • ⭐ sentence-transformers - Multilingual Sentence & Image Embeddings with BERT [GitHub, 18765 stars]

Byte Pair Encoding

  • ⭐ bpemb - Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) [GitHub, 1220 stars]
  • ⭐ subword-nmt - Unsupervised Word Segmentation for Neural Machine Translation and Text Generation [GitHub, 2273 stars]
  • ⭐ python-bpe - Byte Pair Encoding for Python [GitHub, 232 stars]

Transformer-based Architectures

General

Transformer

BERT

Other Transformer Variants

T5
BigBird
Reformer / Linformer / Longformer / Performers
Switch Transformer

GPT-family

General
GPT-3
Learning Resources
Applications
  • ⭐ Awesome GPT-3 - list of all resources related to GPT-3 [GitHub, 4536 stars]
  • 🗂️ GPT-3 Projects - a map of all GPT-3 start-ups and commercial projects
  • 🗂️ GPT-3 Demo Showcase - GPT-3 Demo Showcase, 180+ Apps, Examples, & Resources
  • 🔱 OpenAI API - API Demo to use OpenAI GPT for commercial applications
Open-source Efforts

Other

Distillation, Pruning and Quantization

Reading Material
Tools
  • ⭐ Bert-squeeze - code to reduce the size of Transformer-based models or decrease their latency at inference time [GitHub, 85 stars]
  • ⭐ XtremeDistil - XtremeDistilTransformers for Distilling Massive Multilingual Neural Networks [GitHub, 157 stars]

Automated Summarization

  • 📙 PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization by Google AI [Blog, June 2020]
  • ⭐ CTRLsum - CTRLsum: Towards Generic Controllable Text Summarization [GitHub, 150 stars]
  • ⭐ XL-Sum - XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages [GitHub, 277 stars]
  • ⭐ SummerTime - an open-source text summarization toolkit for non-experts [GitHub, 280 stars]
  • ⭐ PRIMER - PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization [GitHub, 157 stars]
  • ⭐ summarus - Models for automatic abstractive summarization [GitHub, 172 stars]

Knowledge Graphs and NLP

Model Generation

  • ⭐ smolmodels - agentic framework for building ML models from natural language

Small LLMs

  • smollm - 3B parameter language model designed to push the boundaries of small models [GitHub, 3797 stars]

The-NLP-Industry

Note Section keywords: best practices, MLOps

🔙 Back to the Table of Contents

Best Practices for building NLP Projects

MLOps for NLP

MLOps, especially when applied to NLP, is a set of best practices around automating various parts of the workflow when building and deploying NLP pipelines.

In general, MLOps for NLP includes having the following processes in place:

  • Data Versioning - make sure your training, annotation and other types of data are versioned and tracked
  • Experiment Tracking - make sure that all of your experiments are automatically tracked and saved where they can be easily replicated or retraced
  • Model Registry - make sure any neural models you train are versioned and tracked and it is easy to roll back to any of them
  • Automated Testing and Behavioral Testing - besides regular unit and integration tests, you want to have behavioral tests that check for bias or potential adversarial attacks
  • Model Deployment and Serving - automate model deployment, ideally also with zero-downtime deploys like Blue/Green, Canary deploys etc.
  • Data and Model Observability - track data drift, model accuracy drift etc.

Additionally, there are two more components that are not as prevalent for NLP and are mostly used for Computer Vision and other sub-fields of AI:

  • Feature Store - centralized storage of all features developed for ML models than can be easily reused by any other ML project
  • Metadata Management - storage for all information related to the usage of ML models, mainly for reproducing behavior of deployed ML models, artifact tracking etc.

MLOps Compilations & Awesome Lists

Running LLMs locally or self-hosted

  • ⭐ vLLM [GitHub, 81616 stars]
  • ⭐ llama.cpp [GitHub, 114160 stars]
  • 🔱 ollama [Free Local & Paid Cloud Service]

Reading Material

Learning Material

MLOps Communities

Data Versioning

  • ⭐ DVC - Data Version Control (DVC) tracks ML models and data sets [Free and Open Source] Link to GitHub
  • 🔱 Weights & Biases - tools for experiment tracking and dataset versioning [Paid Service]

Experiment Tracking

  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • 🔱 Weights & Biases - tools for experiment tracking and dataset versioning [Paid Service]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]
  • 🔱 SigOpt - automate training & tuning, visualize & compare runs [Paid Service]
  • ⭐ Optuna - hyperparameter optimization framework [GitHub, 14280 stars]
  • ⭐ Clear ML - experiment, orchestrate, deploy, and build data stores, all in one place [Free and Open Source] Link to GitHub
  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
Model Registry
  • ⭐ DVC - Data Version Control (DVC) tracks ML models and data sets [Free and Open Source] Link to GitHub
  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • ⭐ ModelDB - open-source system for Machine Learning model versioning, metadata, and experiment management [GitHub, 1747 stars]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]
  • 🔱 Valohai - End-to-end ML pipelines [Paid Service]
  • 🔱 Pachyderm - version control for data with the tools to build scalable end-to-end ML/AI pipelines [Paid Service with Free Tier]
  • 🔱 polyaxon - reproduce, automate, and scale your data science workflows with production-grade MLOps tools [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]

Automated Testing and Behavioral Testing

  • ⭐ CheckList - Beyond Accuracy: Behavioral Testing of NLP models [GitHub, 2050 stars]
  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ WildNLP - Corrupt an input text to test NLP models' robustness [GitHub, 76 stars]
  • ⭐ Great Expectations - Write tests for your data [GitHub, 11532 stars]
  • ⭐ Deepchecks - Python package for comprehensively validating your machine learning models and data [GitHub, 4017 stars]

Model Deployability and Serving

  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • 🔱 Amazon SageMaker [Paid Service]
  • 🔱 Valohai - End-to-end ML pipelines [Paid Service]
  • 🔱 NLP Cloud - Production-ready NLP API [Paid Service]
  • 🔱 Saturn Cloud [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]
  • 🔱 polyaxon - reproduce, automate, and scale your data science workflows with production-grade MLOps tools [Paid Service]
  • ⭐ TorchServe - flexible and easy to use tool for serving PyTorch models [GitHub, 4359 stars]
  • 🔱 Kubeflow - The Machine Learning Toolkit for Kubernetes [GitHub, 10600 stars]
  • ⭐ KFServing - Serverless Inferencing on Kubernetes [GitHub, 5534 stars]
  • 🔱 TFX - TensorFlow Extended - end-to-end platform for deploying production ML pipelines [Paid Service]
  • 🔱 Pachyderm - version control for data with the tools to build scalable end-to-end ML/AI pipelines [Paid Service with Free Tier]
  • 🔱 Cortex - containers as a service on AWS [Paid Service]
  • 🔱 Azure Machine Learning - end-to-end machine learning lifecycle [Paid Service]
  • ⭐ End2End Serverless Transformers On AWS Lambda [GitHub, 122 stars]
  • ⭐ NLP-Service - sample demo of NLP as a service platform built using FastAPI and Hugging Face [GitHub, 13 stars]
  • 🔱 Dagster - data orchestrator for machine learning [Free and Open Source]
  • 🔱 Verta - AI and machine learning deployment and operations [Paid Service]
  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
  • ⭐ flyte - workflow automation platform for complex, mission-critical data and ML processes at scale [GitHub, 7056 stars]
  • ⭐ MLRun - Machine Learning automation and tracking [GitHub, 1670 stars]
  • 🔱 DataRobot MLOps - DataRobot MLOps provides a center of excellence for your production AI

Model Debugging

  • ⭐ imodels - package for concise, transparent, and accurate predictive modeling [GitHub, 1592 stars]
  • ⭐ Cockpit - A Practical Debugging Tool for Training Deep Neural Networks [GitHub, 488 stars]

Model Accuracy Prediction

  • ⭐ WeightWatcher - WeightWatcher tool for predicting the accuracy of Deep Neural Networks [GitHub, 1751 stars]

Data and Model Observability

General
  • ⭐ Arize AI - embedding drift monitoring for NLP models
  • ⭐ Arize-Phoenix - ML observability for LLMs, vision, language, and tabular models
  • ⭐ whylogs - open source standard for data and ML logging [GitHub, 2819 stars]
  • ⭐ Rubrix - open-source tool for exploring and iterating on data for artificial intelligence projects [GitHub, 4992 stars]
  • ⭐ MLRun - Machine Learning automation and tracking [GitHub, 1670 stars]
  • 🔱 DataRobot MLOps - DataRobot MLOps provides a center of excellence for your production AI
  • 🔱 Cortex - containers as a service on AWS [Paid Service]
Model Centric
  • 🔱 Algorithmia - minimize risk with advanced reporting and enterprise-grade security and governance across all data, models, and infrastructure [Paid Service]
  • 🔱 Dataiku - dataiku is for teams who want to deliver advanced analytics using the latest techniques at big data scale [Paid Service]
  • ⭐ Evidently AI - tools to analyze and monitor machine learning models [Free and Open Source] Link to GitHub
  • 🔱 Fiddler - All-in-one ML and LLM observability. Fastest LLM Guardrails. [Paid Service]
  • 🔱 Hydrosphere - open-source platform for managing ML models [Paid Service]
  • 🔱 Verta - AI and machine learning deployment and operations [Paid Service]
  • 🔱 Domino Model Ops - Deploy and Manage Models to Drive Business Impact [Paid Service]
Data Centric
  • 🔱 Datafold - data quality through diffs, profiling, and anomaly detection [Paid Service]
  • 🔱 acceldata - improve reliability, accelerate scale, and reduce costs across all data pipelines [Paid Service]
  • 🔱 Bigeye - monitoring and alerting to your datasets in minutes [Paid Service]
  • 🔱 datakin - end-to-end, real-time data lineage solution [Paid Service]
  • 🔱 Monte Carlo - data integrity, drifts, schema, lineage [Paid Service]
  • 🔱 SODA - data monitoring, testing and validation [Paid Service]

Feature Stores

  • 🔱 Tecton - enterprise feature store for machine learning [Paid Service]
  • ⭐ FEAST - open source feature store for machine learning Website [GitHub, 7063 stars]
  • 🔱 Hopsworks Feature Store - data management system for managing machine learning features [Paid Service]

Metadata Management

  • ⭐ ML Metadata - a library for recording and retrieving metadata associated with ML developer and data scientist workflows [GitHub, 678 stars]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]

MLOps Frameworks

  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
  • ⭐ kedro - Python framework for creating reproducible, maintainable and modular data science code [GitHub, 10867 stars]
  • ⭐ Seldon Core - MLOps framework to package, deploy, monitor and manage thousands of production machine learning models [GitHub, 4752 stars]
  • ⭐ ZenML - MLOps framework to create reproducible ML pipelines for production machine learning [GitHub, 5429 stars]
  • 🔱 Google Vertex AI - build, deploy, and scale ML models faster, with pre-trained and custom tooling within a unified AI platform [Paid Service]
  • ⭐ Diffgram - Complete training data platform for machine learning delivered as a single application [GitHub, 1904 stars]
  • 🔱 Continual.ai - build, deploy, and operationalize ML models easier and faster with a declarative interface on cloud data warehouses like Snowflake, BigQuery, RedShift, and Databricks. [Paid Service]

Transformer-based Architectures

🔙 Back to the Table of Contents

General

Multi-GPU Transformers
Training Transformers Effectively

Embeddings as a Service

NLP Recipes Industrial Applications:

  • ⭐ Blackstone - A spaCy pipeline and model for NLP on unstructured legal text [GitHub, 689 stars]
  • ⭐ Sci spaCy - spaCy pipeline and models for scientific/biomedical documents [GitHub, 1960 stars]
  • ⭐ FinBERT: Pre-Trained on SEC Filings for Financial NLP Tasks [GitHub, 204 stars]
  • ⭐ LexNLP - Information retrieval and extraction for real, unstructured legal text [GitHub, 781 stars]
  • ⭐ NerDL and NerCRF - Tutorial on Named Entity Recognition for Healthcare with SparkNLP
  • ⭐ Legal Text Analytics - A list of selected resources dedicated to Legal Text Analytics [GitHub, 720 stars]
  • ⭐ BioIE - A curated list of resources relevant to doing Biomedical Information Extraction [GitHub, 441 stars]

The-NLP-Speech

Note Section keywords: speech recognition

🔙 Back to the Table of Contents

General Speech Recognition

  • ⭐ wav2letter - Automatic Speech Recognition Toolkit [GitHub, 6444 stars]
  • ⭐ DeepSpeech - Baidu's DeepSpeech architecture [GitHub, 26750 stars]
  • 📙 Acoustic Word Embeddings by Maria Obedkova [Blog, 2020]
  • ⭐ kaldi - Kaldi is a toolkit for speech recognition [GitHub, 15401 stars]
  • ⭐ awesome-kaldi - resources for using Kaldi [GitHub, 538 stars]
  • ⭐ ESPnet - End-to-End Speech Processing Toolkit [GitHub, 9850 stars]
  • 📙 HuBERT - Self-supervised representation learning for speech recognition, generation, and compression [Blog, June 2021]

Text to Speech / Speech Generation

  • ⭐ FastSpeech - The Implementation of FastSpeech based on pytorch [GitHub, 880 stars]
  • ⭐ TTS - a deep learning toolkit for Text-to-Speech [GitHub, 45454 stars]
  • 🔱 NotebookLM - Google Gemini powered personal assistant / podcast generator

Speech to Text

  • ⭐ whisper - Robust Speech Recognition via Large-Scale Weak Supervision, by OpenAI [GitHub, 101153 stars]
  • ⭐ vibe - GUI tool to work with whisper, multilingual and cuda support included [GitHub, 6317 stars]

Datasets

  • ⭐ VoxPopuli - large-scale multilingual speech corpus for representation learning [GitHub, 573 stars]

The-NLP-Topics

Note Section keywords: topic modeling

🔙 Back to the Table of Contents

Blogs

Frameworks for Topic Modeling

  • ⭐ gensim - framework for topic modeling [GitHub, 16421 stars]
  • ⭐ Spark NLP [GitHub, 4131 stars]

Repositories

Keyword-Extraction

Note Section keywords: keyword extraction

🔙 Back to the Table of Contents

Text Rank

  • ⭐ PyTextRank - PyTextRank is a Python implementation of TextRank as a spaCy pipeline extension [GitHub, 2213 stars]
  • ⭐ textrank - TextRank implementation for Python 3 [GitHub, 1269 stars]

RAKE - Rapid Automatic Keyword Extraction

  • ⭐ rake-nltk - Rapid Automatic Keyword Extraction algorithm using NLTK [GitHub, 1084 stars]
  • ⭐ yake - Single-document unsupervised keyword extraction [GitHub, 1861 stars]
  • ⭐ RAKE-tutorial - A python implementation of the Rapid Automatic Keyword Extraction [GitHub, 372 stars]
  • ⭐ rake-nltk - Rapid Automatic Keyword Extraction algorithm using NLTK [GitHub, 1084 stars]

Other Approaches

  • ⭐ flashtext - Extract Keywords from sentence or Replace keywords in sentences [GitHub, 5714 stars]
  • ⭐ BERT-Keyword-Extractor - Deep Keyphrase Extraction using BERT [GitHub, 261 stars]
  • ⭐ keyBERT - Minimal keyword extraction with BERT [GitHub, 4182 stars]
  • ⭐ KeyphraseVectorizers - vectorizers that extract keyphrases with part-of-speech patterns [GitHub, 267 stars]

Further Reading

Responsible-NLP

Note Section keywords: ethics, responsible NLP

🔙 Back to the Table of Contents

NLP and ML Interpretability

NLP-centric

General

  • ⭐ Language Interpretability Tool (LIT) [GitHub, 3654 stars]
  • ⭐ WhatLies - Toolkit to help visualise - what lies in word embeddings [GitHub, 480 stars]
  • ⭐ Interpret-Text - Interpretability techniques and visualization dashboards for NLP models [GitHub, 432 stars]
  • ⭐ InterpretML - Fit interpretable models. Explain blackbox machine learning [GitHub, 6866 stars]
  • ⭐ thermostat - Collection of NLP model explanations and accompanying analysis tools [GitHub, 143 stars]
  • ⭐ Dodrio - Exploring attention weights in transformer-based models with linguistic knowledge [GitHub, 372 stars]
  • ⭐ imodels - package for concise, transparent, and accurate predictive modeling [GitHub, 1592 stars]

Ethics, Bias, and Equality in NLP

Adversarial Attacks for NLP

Hate Speech Analysis

  • ⭐ HateXplain - BERT for detecting abusive language [GitHub, 240 stars]

NLP & Security

  • ⭐ vuln2vc - domain-specific Word2Vec model for cybersecurity text mining and NLP research [GitHub, 2 stars]

The-NLP-Frameworks

Note Section keywords: frameworks

🔙 Back to the Table of Contents

General Purpose

  • ⭐ spaCy by Explosion AI [GitHub, 33624 stars]
  • ⭐ flair by Zalando [GitHub, 14377 stars]
  • ⭐ AllenNLP by AI2 [GitHub, 11897 stars]
  • ⭐ stanza (former Stanford NLP) [GitHub, 7806 stars]
  • ⭐ spaCy stanza [GitHub, 748 stars]
  • ⭐ nltk [GitHub, 14634 stars]
  • ⭐ gensim - framework for topic modeling [GitHub, 16421 stars]
  • ⭐ pororo - Platform of neural models for natural language processing [GitHub, 1307 stars]
  • ⭐ NLP Architect - A Deep Learning NLP/NLU library by Intel® AI Lab [GitHub, 2933 stars]
  • ⭐ FARM [GitHub, 1754 stars]
  • ⭐ gobbli by RTI International [GitHub, 274 stars]
  • ⭐ headliner - training and deployment of seq2seq models [GitHub, 228 stars]
  • ⭐ SyferText - A privacy preserving NLP framework [GitHub, 198 stars]
  • ⭐ DeText - Text Understanding Framework for Ranking and Classification Tasks [GitHub, 1265 stars]
  • ⭐ TextHero - Text preprocessing, representation and visualization [GitHub, 2911 stars]
  • ⭐ textblob - TextBlob: Simplified Text Processing [GitHub, 9536 stars]
  • ⭐ AdaptNLP - A high level framework and library for NLP [GitHub, 407 stars]
  • ⭐ textacy - NLP, before and after spaCy [GitHub, 2241 stars]
  • ⭐ texar - Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow [GitHub, 2391 stars]
  • ⭐ jiant - jiant is an NLP toolkit [GitHub, 1675 stars]

Data Augmentation

  • ⭐ WildNLP Text manipulation library to test NLP models [GitHub, 76 stars]
  • ⭐ snorkel Framework to generate training data [GitHub, 5970 stars]
  • ⭐ NLPAug Data augmentation for NLP [GitHub, 4657 stars]
  • ⭐ SentAugment Data augmentation by retrieving similar sentences from larger datasets [GitHub, 358 stars]
  • ⭐ faker - Python package that generates fake data for you [GitHub, 19253 stars]
  • ⭐ textflint - Unified Multilingual Robustness Evaluation Toolkit for NLP [GitHub, 651 stars]
  • ⭐ Parrot - Practical and feature-rich paraphrasing framework [GitHub, 919 stars]
  • ⭐ AugLy - data augmentations library for audio, image, text, and video [GitHub, 5084 stars]
  • ⭐ TextAugment - Python 3 library for augmenting text for natural language processing applications [GitHub, 438 stars]

Adversarial NLP Attacks & Behavioral Testing

  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ CleverHans - adversarial example library for constructing NLP attacks and building defenses [GitHub, 6438 stars]
  • ⭐ CheckList - Beyond Accuracy: Behavioral Testing of NLP models [GitHub, 2050 stars]

Transformer-oriented

  • ⭐ transformers by HuggingFace [GitHub, 161166 stars]
  • ⭐ Adapter Hub and its documentation - Adapter modules for Transformers [GitHub, 2812 stars]
  • ⭐ haystack - Transformers at scale for question answering & neural search. [GitHub, 25432 stars]

Dialogue Systems and Speech, Voice Agents

  • ⭐ DeepPavlov by MIPT [GitHub, 6986 stars]
  • ⭐ ParlAI by FAIR [GitHub, 10627 stars]
  • ⭐ rasa - Framework for Conversational Agents [GitHub, 21190 stars]
  • ⭐ wav2letter - Automatic Speech Recognition Toolkit [GitHub, 6444 stars]
  • ⭐ ChatterBot - conversational dialog engine for creating chatbots [GitHub, 14488 stars]
  • ⭐ SpeechBrain - open-source and all-in-one speech toolkit based on PyTorch [GitHub, 11581 stars]
  • ⭐ dialoguefactory Generate continuous dialogue data in a simulated textual world [GitHub, 5 stars]
  • ⭐ gabber AI applications that can see, hear, and speak using your screens, microphones [GitHub, 1103 stars]

Word/Sentence-embeddings oriented

  • ⭐ MUSE A library for Multilingual Unsupervised or Supervised word Embeddings [GitHub, 3246 stars]
  • ⭐ vecmap A framework to learn cross-lingual word embedding mappings [GitHub, 654 stars]
  • ⭐ sentence-transformers - Multilingual Sentence & Image Embeddings with BERT [GitHub, 18765 stars]

Social Media Oriented

  • ⭐ Ekphrasis - text processing tool, geared towards text from social networks [GitHub, 676 stars]

Phonetics

  • ⭐ DeepPhonemizer - grapheme to phoneme conversion with deep learning [GitHub, 426 stars]

Morphology

  • ⭐ LemmInflect - python module for English lemmatization and inflection [GitHub, 280 stars]
  • ⭐ Inflect - generate plurals, ordinals, indefinite articles [GitHub, 1076 stars]
  • ⭐ simplemma - simple multilingual lemmatizer for Python [GitHub, 1076 stars]

Multi-lingual tools

  • ⭐ polyglot - Multi-lingual NLP Framework [GitHub, 2367 stars]
  • ⭐ trankit - Light-Weight Transformer-based Python Toolkit for Multilingual NLP [GitHub, 795 stars]

Distributed NLP / Multi-GPU NLP

Machine Translation

  • ⭐ COMET -A Neural Framework for MT Evaluation [GitHub, 756 stars]
  • ⭐ marian-nmt - Fast Neural Machine Translation in C++ [GitHub, 1449 stars]
  • ⭐ argos-translate - Open source neural machine translation in Python [GitHub, 6092 stars]
  • ⭐ Opus-MT - Open neural machine translation models and web services [GitHub, 819 stars]
  • ⭐ dl-translate - A deep learning-based translation library built on Huggingface transformers [GitHub, 497 stars]
  • ⭐ CTranslate2 - CTranslate2 end-to-end machine translation [GitHub, 4507 stars]

Entity and String Matching

  • ⭐ PolyFuzz - Fuzzy string matching, grouping, and evaluation [GitHub, 797 stars]
  • ⭐ pyahocorasick - Python module implementing Aho-Corasick algorithm for string matching [GitHub, 1104 stars]
  • ⭐ fuzzywuzzy - Fuzzy String Matching in Python [GitHub, 9259 stars]
  • ⭐ jellyfish - approximate and phonetic matching of strings [GitHub, 2215 stars]
  • ⭐ textdistance - Compute distance between sequences [GitHub, 3534 stars]
  • ⭐ DeepMatcher - Compute distance between sequences [GitHub, 617 stars]
  • ⭐ RE2 - Simple and Effective Text Matching with Richer Alignment Features [GitHub, 341 stars]
  • ⭐ Machamp - Machamp: A Generalized Entity Matching Benchmark [GitHub, 21 stars]
  • ⭐ bge-m3 - BGE-M3 hybrid retrieval + re-ranking GitHub [GitHub, 11753 stars]
  • 📙 Cosine_Similarity_Explainer - Semantic Similarity Explainer with AI

Discourse Analysis

  • ⭐ ConvoKit - Cornell Conversational Analysis Toolkit [GitHub, 635 stars]

PII scrubbing

  • ⭐ scrubadub - Clean personally identifiable information from dirty dirty text [GitHub, 425 stars]

Hastag Segmentation

  • ⭐ hashformers - automatically inserting the missing spaces between the words in a hashtag [GitHub, 77 stars]

Books Analysis / Literary Analysis / Semantic Search

  • ⭐ booknlp - a natural language processing pipeline that scales to books and other long documents (in English) [GitHub, 918 stars]
  • ⭐ bookworm - ingests novels, builds an implicit character network and a deeply analysable graph [GitHub, 76 stars]
  • ⭐ SemanticFinder - frontend-only live semantic search with transformers.js [GitHub, 327 stars]

Non-English oriented

Japanese

  • ⭐ fugashi - Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis [GitHub, 521 stars]
  • ⭐ SudachiPy - SudachiPy is a Python version of Sudachi, a Japanese morphological analyzer [GitHub, 437 stars]
  • ⭐ Konoha - easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code [GitHub, 261 stars]
  • ⭐ jProcessing - Japanese Natural Langauge Processing Libraries [GitHub, 147 stars]
  • ⭐ Ginza - Japanese NLP Library using spaCy as framework based on Universal Dependencies [GitHub, 854 stars]
  • ⭐ kuromoji - self-contained and very easy to use Japanese morphological analyzer designed for search [GitHub, 1043 stars]
  • ⭐ nagisa - Japanese tokenizer based on recurrent neural networks [GitHub, 417 stars]
  • ⭐ KyTea - Kyoto Text Analysis Toolkit for word segmentation and pronunciation estimation [GitHub, 213 stars]
  • ⭐ Jigg - Pipeline framework for easy natural language processing [GitHub, 77 stars]
  • ⭐ Juman++ - Juman++ (a Morphological Analyzer Toolkit) [GitHub, 414 stars]
  • ⭐ RakutenMA - morphological analyzer (word segmentor + PoS Tagger) for Chinese and Japanese written purely in JavaScript [GitHub, 472 stars]
  • ⭐ toiro - a comparison tool of Japanese tokenizers [GitHub, 123 stars]

Thai

  • ⭐ AttaCut - Fast and Reasonably Accurate Word Tokenizer for Thai [GitHub, 96 stars]
  • ⭐ ThaiLMCut - Word Tokenizer for Thai Language [GitHub, 17 stars]

Chinese

  • ⭐ Spacy-pkuseg - The pkuseg toolkit for multi-domain Chinese word segmentation [GitHub, 70 stars]

Ukrainian

  • ⭐ recruitment-dataset - Recruitment Dataset Preprocessing and Recommender System (Ukrainian, English)

Other

  • ⭐ textblob-de - TextBlob: Simplified Text Processing for German [GitHub, 103 stars]
  • ⭐ Kashgari Transfer Learning with focus on Chinese [GitHub, 2384 stars]
  • ⭐ Underthesea - Vietnamese NLP Toolkit [GitHub, 1737 stars]
  • ⭐ PTT5 - Pretraining and validating the T5 model on Brazilian Portuguese data [GitHub, 91 stars]

Text Data Labelling & Classification

  • ⭐ Small-Text - Active Learning for Text Classifcation in Python [GitHub, 643 stars]
  • ⭐ Doccano - open source annotation tool for machine learning practitioners [GitHub, 10659 stars]
  • ⭐ Adala - Autonomous DAta (Labeling) Agent framework [GitHub, 1593 stars]
  • ⭐ EDA - Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks [GitHub, 1652 stars]
  • 🔱 Prodigy - annotation tool powered by active learning [Paid Service]

The-NLP-Learning

Note Section keywords: learn NLP

🔙 Back to the Table of Contents

General

Courses

Books

Tutorials

The-NLP-Communities

Other-NLP-Topics

🔙 Back to the Table of Contents

Tokenization

  • ⭐ tokenizers - Fast State-of-the-Art Tokenizers optimized for Research and Production [GitHub, 10782 stars]
  • ⭐ SentencePiece - Unsupervised text tokenizer for Neural Network-based text generation [GitHub, 11870 stars]
  • ⭐ SoMaJo - A tokenizer and sentence splitter for German and English web and social media texts [GitHub, 152 stars]

Data Augmentation and Weak Supervision

Libraries and Frameworks
  • ⭐ WildNLP Text manipulation library to test NLP models [GitHub, 76 stars]
  • ⭐ NLPAug Data augmentation for NLP [GitHub, 4657 stars]
  • ⭐ SentAugment Data augmentation by retrieving similar sentences from larger datasets [GitHub, 358 stars]
  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ skweak - software toolkit for weak supervision applied to NLP tasks [GitHub, 927 stars]
  • ⭐ NL-Augmenter - Collaborative Repository of Natural Language Transformations [GitHub, 787 stars]
  • ⭐ EDA - Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks [GitHub, 1652 stars]
  • ⭐ snorkel Framework to generate training data [GitHub, 5970 stars]
  • ⭐ dialoguefactory Generate continuous dialogue data in a simulated textual world [GitHub, 5 stars]
Reading Material and Tutorials

Named Entity Recognition (NER)

Relation Extraction

  • ⭐ tacred-relation TACRED: position-aware attention model for relation extraction [GitHub, 359 stars]
  • ⭐ tacrev TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [GitHub, 70 stars]
  • ⭐ tac-self-attention Relation extraction with position-aware self-attention [GitHub, 66 stars]
  • ⭐ Re-TACRED Re-TACRED: Addressing Shortcomings of the TACRED Dataset [GitHub, 55 stars]

Coreference Resolution

Sentiment Analysis

Domain Adaptation

Low Resource NLP

Spell Correction / Error Correction

  • ⭐ Gramformer - ramework for detecting, highlighting and correcting grammatical errors [GitHub, 1581 stars]
  • ⭐ NeuSpell - A Neural Spelling Correction Toolkit [GitHub, 712 stars]
  • ⭐ SymSpellPy - Python port of SymSpell [GitHub, 871 stars]
  • 📙 Speller100 by Microsoft [Blog, Feb 2021]
  • ⭐ JamSpell - spell checking library - accurate, fast, multi-language [GitHub, 662 stars]
  • ⭐ pycorrector - spell correction for Chinese [GitHub, 6452 stars]
  • ⭐ contractions - Fixes contractions such as you're to you are [GitHub, 318 stars]
  • 📙 Fine Tuning T5 for Grammar Correction by Sachin Abeywardana [Blog, Nov 2022]

PDF Parsing

  • ⭐ spacy-layout - Process PDFs, Word documents and more with spaCy [GitHub, 902 stars]
  • ⭐ bentopdf - A Privacy First PDF Toolkit [GitHub, 13552 stars]

Style Transfer for NLP

  • ⭐ Styleformer - Neural Language Style Transfer framework [GitHub, 494 stars]
  • ⭐ StylePTB - A Compositional Benchmark for Fine-grained Controllable Text Style Transfer [GitHub, 64 stars]

Automata Theory for NLP

  • ⭐ pyahocorasick - Python module implementing Aho-Corasick algorithm for string matching [GitHub, 1104 stars]

Obscene words detection

  • ⭐ LDNOOBW - List of Dirty, Naughty, Obscene, and Otherwise Bad Words [GitHub, 3376 stars]

Reddit Analysis

  • ⭐ Subreddit Analyzer - comprehensive Data and Text Mining workflow for submissions and comments from any given public subreddit [GitHub, 499 stars]

Skill Detection

  • ⭐ SkillNER - rule based NLP module to extract job skills from text [GitHub, 209 stars]

Reinforcement Learning for NLP

  • ⭐ nlp-gym - NLPGym - A toolkit to develop RL agents to solve NLP tasks [GitHub, 201 stars]

AutoML / AutoNLP

  • ⭐ AutoNLP - Faster and easier training and deployments of SOTA NLP models [GitHub, 4574 stars]
  • ⭐ TPOT - Python Automated Machine Learning tool [GitHub, 10047 stars]
  • ⭐ Auto-PyTorch - Automatic architecture search and hyperparameter optimization for PyTorch [GitHub, 2534 stars]
  • ⭐ HungaBunga - Brute-Force all sklearn models with all parameters using .fit .predict [GitHub, 0 stars]
  • ⭐ Optuna - hyperparameter optimization framework [GitHub, 14280 stars]
  • ⭐ FLAML - fast and lightweight AutoML library [GitHub, 4360 stars]
  • ⭐ Gradsflow - open-source AutoML & PyTorch Model Training Library [GitHub, 307 stars]

OCR - Optical Character Recognition

Document AI

Text Generation

Title / Headlines Generation

  • ⭐ TitleStylist Learning to Generate Headlines with Controlled Styles [GitHub, 78 stars]

NLP research reproducibility

License CC0

Attributions

Resources

  • All linked resources belong to original authors

Icons

Fonts

The Pandect Series also includes

     

awesome-list
deeplearning
natural-language-processing
naturallanguageprocessing
nlp
pandect

Contributors

ivan-bilan

156 commits

axiomofjoy

2 commits

anoopkunchukuttan

2 commits

Darshnadvani

1 commits

Similar repos

keon/awesome-nlp

:book: A curated list of resources dedicated to Natural Language Processing (NLP)

19,028

ElizaLo/NLP-Natural-Language-Processing

Projects and useful articles / links

435

fighting41love/funNLP

中英文敏感词、语言检测、中外手机/电话归属地/运营商查询、名字推断性别、手机号抽取、身份证抽取、邮箱抽取、中日文人名库、中文缩写库、拆字词典、词汇情感值、停用词、反动词表、暴恐词表、繁简体转换、英文…

83,345

changwookjun/nlp-paper

NLP Paper

592

NiuTrans/ABigSurvey

A collection of 1000+ survey papers on Natural Language Processing (NLP) and Machine Learning (ML).

2,028

Kyubyong/nlp_tasks

Natural Language Processing Tasks and References

3,008

accelerated-text/awesome-nlg

A curated list of resources dedicated to Natural Language Generation (NLG)

483

cedrickchee/awesome-transformer-nlp

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT,…

1,151

Languages

Python

100.0%

ivan-bilan/The-NLP-Pandect

A comprehensive reference for all topics related to Natural Language Processing

Python

2,039

167 commits

updated Jun 1, 2026

See the code

README

The-NLP-Pandect

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find almost anything related to Natural Language Processing that is available online.

Note Quick legend on available resource types:

⭐ - open source project, usually a GitHub repository with its number of stars

📙 - resource you can read, usually a blog post or a paper

🗂️ - a collection of additional resources

🔱 - non-open source tool, framework or paid service

🎥️ - a resource you can watch

🎙️ - a resource you can listen to

Table of Contents

📇 Main Section🗃️ Sub-sections Sample
NLP ResourcesPaper Summaries, Conference Summaries, NLP Datasets
NLP PodcastsNLP-only Podcasts, Podcasts with many NLP Episodes
NLP Newsletters-
NLP Meetups-
NLP YouTube Channels-
NLP BenchmarksGeneral NLU, Question Answering, Multilingual
Research ResourcesResource on Transformer Models, Distillation and Pruning, Automated Summarization
Industry ResourcesBest Practices for NLP Systems, MLOps for NLP
Speech RecognitionGeneral Resources, Text to Speech, Speech to Text, Datasets
Topic ModelingBlogs, Frameworks, Repositories and Projects
Keyword ExtractionText Rank, Rake, Other Approaches
Responsible NLPNLP and ML Interpretability, Ethics, Bias, and Equality in NLP, Adversarial Attacks for NLP
NLP FrameworksGeneral Purpose, Data Augmentation, Machine Translation, Adversarial Attacks, Dialog Systems & Speech, Entity and String Matching, Non-English Frameworks, Text Annotation
Learning NLPCourses, Books, Tutorials
NLP Communities-
Other NLP TopicsTokenization, Data Augmentation, Named Entity Recognition, Error Correction, AutoML/AutoNLP, Text Generation

The-NLP-Resources

Note Section keywords: paper summaries, compendium, awesome list

Compendiums and awesome lists on the topic of NLP:

NLP Conferences, Paper Summaries and Paper Compendiums:

Papers and Paper Summaries
Conference Summaries

NLP Progress and NLP Tasks:

NLP Datasets:

Word and Sentence embeddings:

Notebooks, Scripts and Repositories

Non-English resources and Compendiums

Pre-trained NLP models

NLP History

General
2020 Year in Review

The-NLP-Podcasts

🔙 Back to the Table of Contents

NLP-only podcasts

Many NLP episodes

Some NLP episodes

The-NLP-Newsletter

The-NLP-Meetups

The-NLP-Youtube

The-NLP-Benchmarks

🔙 Back to the Table of Contents

General NLU

  • ⭐ GLUE - General Language Understanding Evaluation (GLUE) benchmark
  • ⭐ SuperGLUE - benchmark styled after GLUE with a new set of more difficult language understanding tasks
  • ⭐ decaNLP - The Natural Language Decathlon (decaNLP) for studying general NLP models
  • ⭐ dialoglue - DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue [GitHub, 287 stars]
  • ⭐ DynaBench - Dynabench is a research platform for dynamic data collection and benchmarking
  • ⭐ Big-Bench - collaborative benchmark for measuring and extrapolating the capabilities of language models [GitHub, 3244 stars]

Summarization

  • ⭐ WikiAsp - WikiAsp: Multi-document aspect-based summarization Dataset
  • ⭐ WikiLingua - A Multilingual Abstractive Summarization Dataset

Question Answering

  • ⭐ SQuAD - Stanford Question Answering Dataset (SQuAD)
  • ⭐ XQuad - XQuAD (Cross-lingual Question Answering Dataset) for cross-lingual question answering
  • ⭐ GrailQA - Strongly Generalizable Question Answering (GrailQA)
  • ⭐ CSQA - Complex Sequential Question Answering

Multilingual and Non-English Benchmarks

  • 📙 XTREME - Massively Multilingual Multi-task Benchmark
  • ⭐ GLUECoS - A benchmark for code-switched NLP
  • ⭐ IndicGLUE - Natural Language Understanding Benchmark for Indic Languages
  • ⭐ LinCE - Linguistic Code-Switching Evaluation Benchmark
  • ⭐ Russian SuperGlue - Russian SuperGlue Benchmark

Bio, Law, and other scientific domains

  • ⭐ BLURB - Biomedical Language Understanding and Reasoning Benchmark
  • ⭐ BLUE - Biomedical Language Understanding Evaluation benchmark
  • ⭐ LexGLUE - A Benchmark Dataset for Legal Language Understanding in English

Transformer Efficiency

Other

  • ⭐ CodeXGLUE - A benchmark dataset for code intelligence
  • ⭐ CrossNER - CrossNER: Evaluating Cross-Domain Named Entity Recognition
  • ⭐ MultiNLI - Multi-Genre Natural Language Inference corpus
  • ⭐ iSarcasm: A Dataset of Intended Sarcasm - iSarcasm is a dataset of tweets, each labelled as either sarcastic or non_sarcastic
  • ⭐ SLTev - tool for comprehensive evaluation of (simultaneous) spoken language translation [GitHub, 12 stars]

The-NLP-Research

🔙 Back to the Table of Contents

General

Embeddings

Repositories

Blogs

Cross-lingual Word and Sentence Embeddings

  • ⭐ vecmap - VecMap (cross-lingual word embedding mappings) [GitHub, 654 stars]
  • ⭐ sentence-transformers - Multilingual Sentence & Image Embeddings with BERT [GitHub, 18765 stars]

Byte Pair Encoding

  • ⭐ bpemb - Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) [GitHub, 1220 stars]
  • ⭐ subword-nmt - Unsupervised Word Segmentation for Neural Machine Translation and Text Generation [GitHub, 2273 stars]
  • ⭐ python-bpe - Byte Pair Encoding for Python [GitHub, 232 stars]

Transformer-based Architectures

General

Transformer

BERT

Other Transformer Variants

T5
BigBird
Reformer / Linformer / Longformer / Performers
Switch Transformer

GPT-family

General
GPT-3
Learning Resources
Applications
  • ⭐ Awesome GPT-3 - list of all resources related to GPT-3 [GitHub, 4536 stars]
  • 🗂️ GPT-3 Projects - a map of all GPT-3 start-ups and commercial projects
  • 🗂️ GPT-3 Demo Showcase - GPT-3 Demo Showcase, 180+ Apps, Examples, & Resources
  • 🔱 OpenAI API - API Demo to use OpenAI GPT for commercial applications
Open-source Efforts

Other

Distillation, Pruning and Quantization

Reading Material
Tools
  • ⭐ Bert-squeeze - code to reduce the size of Transformer-based models or decrease their latency at inference time [GitHub, 85 stars]
  • ⭐ XtremeDistil - XtremeDistilTransformers for Distilling Massive Multilingual Neural Networks [GitHub, 157 stars]

Automated Summarization

  • 📙 PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization by Google AI [Blog, June 2020]
  • ⭐ CTRLsum - CTRLsum: Towards Generic Controllable Text Summarization [GitHub, 150 stars]
  • ⭐ XL-Sum - XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages [GitHub, 277 stars]
  • ⭐ SummerTime - an open-source text summarization toolkit for non-experts [GitHub, 280 stars]
  • ⭐ PRIMER - PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization [GitHub, 157 stars]
  • ⭐ summarus - Models for automatic abstractive summarization [GitHub, 172 stars]

Knowledge Graphs and NLP

Model Generation

  • ⭐ smolmodels - agentic framework for building ML models from natural language

Small LLMs

  • smollm - 3B parameter language model designed to push the boundaries of small models [GitHub, 3797 stars]

The-NLP-Industry

Note Section keywords: best practices, MLOps

🔙 Back to the Table of Contents

Best Practices for building NLP Projects

MLOps for NLP

MLOps, especially when applied to NLP, is a set of best practices around automating various parts of the workflow when building and deploying NLP pipelines.

In general, MLOps for NLP includes having the following processes in place:

  • Data Versioning - make sure your training, annotation and other types of data are versioned and tracked
  • Experiment Tracking - make sure that all of your experiments are automatically tracked and saved where they can be easily replicated or retraced
  • Model Registry - make sure any neural models you train are versioned and tracked and it is easy to roll back to any of them
  • Automated Testing and Behavioral Testing - besides regular unit and integration tests, you want to have behavioral tests that check for bias or potential adversarial attacks
  • Model Deployment and Serving - automate model deployment, ideally also with zero-downtime deploys like Blue/Green, Canary deploys etc.
  • Data and Model Observability - track data drift, model accuracy drift etc.

Additionally, there are two more components that are not as prevalent for NLP and are mostly used for Computer Vision and other sub-fields of AI:

  • Feature Store - centralized storage of all features developed for ML models than can be easily reused by any other ML project
  • Metadata Management - storage for all information related to the usage of ML models, mainly for reproducing behavior of deployed ML models, artifact tracking etc.

MLOps Compilations & Awesome Lists

Running LLMs locally or self-hosted

  • ⭐ vLLM [GitHub, 81616 stars]
  • ⭐ llama.cpp [GitHub, 114160 stars]
  • 🔱 ollama [Free Local & Paid Cloud Service]

Reading Material

Learning Material

MLOps Communities

Data Versioning

  • ⭐ DVC - Data Version Control (DVC) tracks ML models and data sets [Free and Open Source] Link to GitHub
  • 🔱 Weights & Biases - tools for experiment tracking and dataset versioning [Paid Service]

Experiment Tracking

  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • 🔱 Weights & Biases - tools for experiment tracking and dataset versioning [Paid Service]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]
  • 🔱 SigOpt - automate training & tuning, visualize & compare runs [Paid Service]
  • ⭐ Optuna - hyperparameter optimization framework [GitHub, 14280 stars]
  • ⭐ Clear ML - experiment, orchestrate, deploy, and build data stores, all in one place [Free and Open Source] Link to GitHub
  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
Model Registry
  • ⭐ DVC - Data Version Control (DVC) tracks ML models and data sets [Free and Open Source] Link to GitHub
  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • ⭐ ModelDB - open-source system for Machine Learning model versioning, metadata, and experiment management [GitHub, 1747 stars]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]
  • 🔱 Valohai - End-to-end ML pipelines [Paid Service]
  • 🔱 Pachyderm - version control for data with the tools to build scalable end-to-end ML/AI pipelines [Paid Service with Free Tier]
  • 🔱 polyaxon - reproduce, automate, and scale your data science workflows with production-grade MLOps tools [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]

Automated Testing and Behavioral Testing

  • ⭐ CheckList - Beyond Accuracy: Behavioral Testing of NLP models [GitHub, 2050 stars]
  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ WildNLP - Corrupt an input text to test NLP models' robustness [GitHub, 76 stars]
  • ⭐ Great Expectations - Write tests for your data [GitHub, 11532 stars]
  • ⭐ Deepchecks - Python package for comprehensively validating your machine learning models and data [GitHub, 4017 stars]

Model Deployability and Serving

  • ⭐ mlflow - open source platform for the machine learning lifecycle [Free and Open Source] Link to GitHub
  • 🔱 Amazon SageMaker [Paid Service]
  • 🔱 Valohai - End-to-end ML pipelines [Paid Service]
  • 🔱 NLP Cloud - Production-ready NLP API [Paid Service]
  • 🔱 Saturn Cloud [Paid Service]
  • 🔱 Comet ML - enables data scientists and teams to track, compare, explain and optimize experiments and models [Paid Service]
  • 🔱 polyaxon - reproduce, automate, and scale your data science workflows with production-grade MLOps tools [Paid Service]
  • ⭐ TorchServe - flexible and easy to use tool for serving PyTorch models [GitHub, 4359 stars]
  • 🔱 Kubeflow - The Machine Learning Toolkit for Kubernetes [GitHub, 10600 stars]
  • ⭐ KFServing - Serverless Inferencing on Kubernetes [GitHub, 5534 stars]
  • 🔱 TFX - TensorFlow Extended - end-to-end platform for deploying production ML pipelines [Paid Service]
  • 🔱 Pachyderm - version control for data with the tools to build scalable end-to-end ML/AI pipelines [Paid Service with Free Tier]
  • 🔱 Cortex - containers as a service on AWS [Paid Service]
  • 🔱 Azure Machine Learning - end-to-end machine learning lifecycle [Paid Service]
  • ⭐ End2End Serverless Transformers On AWS Lambda [GitHub, 122 stars]
  • ⭐ NLP-Service - sample demo of NLP as a service platform built using FastAPI and Hugging Face [GitHub, 13 stars]
  • 🔱 Dagster - data orchestrator for machine learning [Free and Open Source]
  • 🔱 Verta - AI and machine learning deployment and operations [Paid Service]
  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
  • ⭐ flyte - workflow automation platform for complex, mission-critical data and ML processes at scale [GitHub, 7056 stars]
  • ⭐ MLRun - Machine Learning automation and tracking [GitHub, 1670 stars]
  • 🔱 DataRobot MLOps - DataRobot MLOps provides a center of excellence for your production AI

Model Debugging

  • ⭐ imodels - package for concise, transparent, and accurate predictive modeling [GitHub, 1592 stars]
  • ⭐ Cockpit - A Practical Debugging Tool for Training Deep Neural Networks [GitHub, 488 stars]

Model Accuracy Prediction

  • ⭐ WeightWatcher - WeightWatcher tool for predicting the accuracy of Deep Neural Networks [GitHub, 1751 stars]

Data and Model Observability

General
  • ⭐ Arize AI - embedding drift monitoring for NLP models
  • ⭐ Arize-Phoenix - ML observability for LLMs, vision, language, and tabular models
  • ⭐ whylogs - open source standard for data and ML logging [GitHub, 2819 stars]
  • ⭐ Rubrix - open-source tool for exploring and iterating on data for artificial intelligence projects [GitHub, 4992 stars]
  • ⭐ MLRun - Machine Learning automation and tracking [GitHub, 1670 stars]
  • 🔱 DataRobot MLOps - DataRobot MLOps provides a center of excellence for your production AI
  • 🔱 Cortex - containers as a service on AWS [Paid Service]
Model Centric
  • 🔱 Algorithmia - minimize risk with advanced reporting and enterprise-grade security and governance across all data, models, and infrastructure [Paid Service]
  • 🔱 Dataiku - dataiku is for teams who want to deliver advanced analytics using the latest techniques at big data scale [Paid Service]
  • ⭐ Evidently AI - tools to analyze and monitor machine learning models [Free and Open Source] Link to GitHub
  • 🔱 Fiddler - All-in-one ML and LLM observability. Fastest LLM Guardrails. [Paid Service]
  • 🔱 Hydrosphere - open-source platform for managing ML models [Paid Service]
  • 🔱 Verta - AI and machine learning deployment and operations [Paid Service]
  • 🔱 Domino Model Ops - Deploy and Manage Models to Drive Business Impact [Paid Service]
Data Centric
  • 🔱 Datafold - data quality through diffs, profiling, and anomaly detection [Paid Service]
  • 🔱 acceldata - improve reliability, accelerate scale, and reduce costs across all data pipelines [Paid Service]
  • 🔱 Bigeye - monitoring and alerting to your datasets in minutes [Paid Service]
  • 🔱 datakin - end-to-end, real-time data lineage solution [Paid Service]
  • 🔱 Monte Carlo - data integrity, drifts, schema, lineage [Paid Service]
  • 🔱 SODA - data monitoring, testing and validation [Paid Service]

Feature Stores

  • 🔱 Tecton - enterprise feature store for machine learning [Paid Service]
  • ⭐ FEAST - open source feature store for machine learning Website [GitHub, 7063 stars]
  • 🔱 Hopsworks Feature Store - data management system for managing machine learning features [Paid Service]

Metadata Management

  • ⭐ ML Metadata - a library for recording and retrieving metadata associated with ML developer and data scientist workflows [GitHub, 678 stars]
  • 🔱 Neptune AI - experiment tracking and model registry built for research and production teams [Paid Service]

MLOps Frameworks

  • ⭐ Metaflow - human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects [GitHub, 10111 stars]
  • ⭐ kedro - Python framework for creating reproducible, maintainable and modular data science code [GitHub, 10867 stars]
  • ⭐ Seldon Core - MLOps framework to package, deploy, monitor and manage thousands of production machine learning models [GitHub, 4752 stars]
  • ⭐ ZenML - MLOps framework to create reproducible ML pipelines for production machine learning [GitHub, 5429 stars]
  • 🔱 Google Vertex AI - build, deploy, and scale ML models faster, with pre-trained and custom tooling within a unified AI platform [Paid Service]
  • ⭐ Diffgram - Complete training data platform for machine learning delivered as a single application [GitHub, 1904 stars]
  • 🔱 Continual.ai - build, deploy, and operationalize ML models easier and faster with a declarative interface on cloud data warehouses like Snowflake, BigQuery, RedShift, and Databricks. [Paid Service]

Transformer-based Architectures

🔙 Back to the Table of Contents

General

Multi-GPU Transformers
Training Transformers Effectively

Embeddings as a Service

NLP Recipes Industrial Applications:

  • ⭐ Blackstone - A spaCy pipeline and model for NLP on unstructured legal text [GitHub, 689 stars]
  • ⭐ Sci spaCy - spaCy pipeline and models for scientific/biomedical documents [GitHub, 1960 stars]
  • ⭐ FinBERT: Pre-Trained on SEC Filings for Financial NLP Tasks [GitHub, 204 stars]
  • ⭐ LexNLP - Information retrieval and extraction for real, unstructured legal text [GitHub, 781 stars]
  • ⭐ NerDL and NerCRF - Tutorial on Named Entity Recognition for Healthcare with SparkNLP
  • ⭐ Legal Text Analytics - A list of selected resources dedicated to Legal Text Analytics [GitHub, 720 stars]
  • ⭐ BioIE - A curated list of resources relevant to doing Biomedical Information Extraction [GitHub, 441 stars]

The-NLP-Speech

Note Section keywords: speech recognition

🔙 Back to the Table of Contents

General Speech Recognition

  • ⭐ wav2letter - Automatic Speech Recognition Toolkit [GitHub, 6444 stars]
  • ⭐ DeepSpeech - Baidu's DeepSpeech architecture [GitHub, 26750 stars]
  • 📙 Acoustic Word Embeddings by Maria Obedkova [Blog, 2020]
  • ⭐ kaldi - Kaldi is a toolkit for speech recognition [GitHub, 15401 stars]
  • ⭐ awesome-kaldi - resources for using Kaldi [GitHub, 538 stars]
  • ⭐ ESPnet - End-to-End Speech Processing Toolkit [GitHub, 9850 stars]
  • 📙 HuBERT - Self-supervised representation learning for speech recognition, generation, and compression [Blog, June 2021]

Text to Speech / Speech Generation

  • ⭐ FastSpeech - The Implementation of FastSpeech based on pytorch [GitHub, 880 stars]
  • ⭐ TTS - a deep learning toolkit for Text-to-Speech [GitHub, 45454 stars]
  • 🔱 NotebookLM - Google Gemini powered personal assistant / podcast generator

Speech to Text

  • ⭐ whisper - Robust Speech Recognition via Large-Scale Weak Supervision, by OpenAI [GitHub, 101153 stars]
  • ⭐ vibe - GUI tool to work with whisper, multilingual and cuda support included [GitHub, 6317 stars]

Datasets

  • ⭐ VoxPopuli - large-scale multilingual speech corpus for representation learning [GitHub, 573 stars]

The-NLP-Topics

Note Section keywords: topic modeling

🔙 Back to the Table of Contents

Blogs

Frameworks for Topic Modeling

  • ⭐ gensim - framework for topic modeling [GitHub, 16421 stars]
  • ⭐ Spark NLP [GitHub, 4131 stars]

Repositories

Keyword-Extraction

Note Section keywords: keyword extraction

🔙 Back to the Table of Contents

Text Rank

  • ⭐ PyTextRank - PyTextRank is a Python implementation of TextRank as a spaCy pipeline extension [GitHub, 2213 stars]
  • ⭐ textrank - TextRank implementation for Python 3 [GitHub, 1269 stars]

RAKE - Rapid Automatic Keyword Extraction

  • ⭐ rake-nltk - Rapid Automatic Keyword Extraction algorithm using NLTK [GitHub, 1084 stars]
  • ⭐ yake - Single-document unsupervised keyword extraction [GitHub, 1861 stars]
  • ⭐ RAKE-tutorial - A python implementation of the Rapid Automatic Keyword Extraction [GitHub, 372 stars]
  • ⭐ rake-nltk - Rapid Automatic Keyword Extraction algorithm using NLTK [GitHub, 1084 stars]

Other Approaches

  • ⭐ flashtext - Extract Keywords from sentence or Replace keywords in sentences [GitHub, 5714 stars]
  • ⭐ BERT-Keyword-Extractor - Deep Keyphrase Extraction using BERT [GitHub, 261 stars]
  • ⭐ keyBERT - Minimal keyword extraction with BERT [GitHub, 4182 stars]
  • ⭐ KeyphraseVectorizers - vectorizers that extract keyphrases with part-of-speech patterns [GitHub, 267 stars]

Further Reading

Responsible-NLP

Note Section keywords: ethics, responsible NLP

🔙 Back to the Table of Contents

NLP and ML Interpretability

NLP-centric

General

  • ⭐ Language Interpretability Tool (LIT) [GitHub, 3654 stars]
  • ⭐ WhatLies - Toolkit to help visualise - what lies in word embeddings [GitHub, 480 stars]
  • ⭐ Interpret-Text - Interpretability techniques and visualization dashboards for NLP models [GitHub, 432 stars]
  • ⭐ InterpretML - Fit interpretable models. Explain blackbox machine learning [GitHub, 6866 stars]
  • ⭐ thermostat - Collection of NLP model explanations and accompanying analysis tools [GitHub, 143 stars]
  • ⭐ Dodrio - Exploring attention weights in transformer-based models with linguistic knowledge [GitHub, 372 stars]
  • ⭐ imodels - package for concise, transparent, and accurate predictive modeling [GitHub, 1592 stars]

Ethics, Bias, and Equality in NLP

Adversarial Attacks for NLP

Hate Speech Analysis

  • ⭐ HateXplain - BERT for detecting abusive language [GitHub, 240 stars]

NLP & Security

  • ⭐ vuln2vc - domain-specific Word2Vec model for cybersecurity text mining and NLP research [GitHub, 2 stars]

The-NLP-Frameworks

Note Section keywords: frameworks

🔙 Back to the Table of Contents

General Purpose

  • ⭐ spaCy by Explosion AI [GitHub, 33624 stars]
  • ⭐ flair by Zalando [GitHub, 14377 stars]
  • ⭐ AllenNLP by AI2 [GitHub, 11897 stars]
  • ⭐ stanza (former Stanford NLP) [GitHub, 7806 stars]
  • ⭐ spaCy stanza [GitHub, 748 stars]
  • ⭐ nltk [GitHub, 14634 stars]
  • ⭐ gensim - framework for topic modeling [GitHub, 16421 stars]
  • ⭐ pororo - Platform of neural models for natural language processing [GitHub, 1307 stars]
  • ⭐ NLP Architect - A Deep Learning NLP/NLU library by Intel® AI Lab [GitHub, 2933 stars]
  • ⭐ FARM [GitHub, 1754 stars]
  • ⭐ gobbli by RTI International [GitHub, 274 stars]
  • ⭐ headliner - training and deployment of seq2seq models [GitHub, 228 stars]
  • ⭐ SyferText - A privacy preserving NLP framework [GitHub, 198 stars]
  • ⭐ DeText - Text Understanding Framework for Ranking and Classification Tasks [GitHub, 1265 stars]
  • ⭐ TextHero - Text preprocessing, representation and visualization [GitHub, 2911 stars]
  • ⭐ textblob - TextBlob: Simplified Text Processing [GitHub, 9536 stars]
  • ⭐ AdaptNLP - A high level framework and library for NLP [GitHub, 407 stars]
  • ⭐ textacy - NLP, before and after spaCy [GitHub, 2241 stars]
  • ⭐ texar - Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow [GitHub, 2391 stars]
  • ⭐ jiant - jiant is an NLP toolkit [GitHub, 1675 stars]

Data Augmentation

  • ⭐ WildNLP Text manipulation library to test NLP models [GitHub, 76 stars]
  • ⭐ snorkel Framework to generate training data [GitHub, 5970 stars]
  • ⭐ NLPAug Data augmentation for NLP [GitHub, 4657 stars]
  • ⭐ SentAugment Data augmentation by retrieving similar sentences from larger datasets [GitHub, 358 stars]
  • ⭐ faker - Python package that generates fake data for you [GitHub, 19253 stars]
  • ⭐ textflint - Unified Multilingual Robustness Evaluation Toolkit for NLP [GitHub, 651 stars]
  • ⭐ Parrot - Practical and feature-rich paraphrasing framework [GitHub, 919 stars]
  • ⭐ AugLy - data augmentations library for audio, image, text, and video [GitHub, 5084 stars]
  • ⭐ TextAugment - Python 3 library for augmenting text for natural language processing applications [GitHub, 438 stars]

Adversarial NLP Attacks & Behavioral Testing

  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ CleverHans - adversarial example library for constructing NLP attacks and building defenses [GitHub, 6438 stars]
  • ⭐ CheckList - Beyond Accuracy: Behavioral Testing of NLP models [GitHub, 2050 stars]

Transformer-oriented

  • ⭐ transformers by HuggingFace [GitHub, 161166 stars]
  • ⭐ Adapter Hub and its documentation - Adapter modules for Transformers [GitHub, 2812 stars]
  • ⭐ haystack - Transformers at scale for question answering & neural search. [GitHub, 25432 stars]

Dialogue Systems and Speech, Voice Agents

  • ⭐ DeepPavlov by MIPT [GitHub, 6986 stars]
  • ⭐ ParlAI by FAIR [GitHub, 10627 stars]
  • ⭐ rasa - Framework for Conversational Agents [GitHub, 21190 stars]
  • ⭐ wav2letter - Automatic Speech Recognition Toolkit [GitHub, 6444 stars]
  • ⭐ ChatterBot - conversational dialog engine for creating chatbots [GitHub, 14488 stars]
  • ⭐ SpeechBrain - open-source and all-in-one speech toolkit based on PyTorch [GitHub, 11581 stars]
  • ⭐ dialoguefactory Generate continuous dialogue data in a simulated textual world [GitHub, 5 stars]
  • ⭐ gabber AI applications that can see, hear, and speak using your screens, microphones [GitHub, 1103 stars]

Word/Sentence-embeddings oriented

  • ⭐ MUSE A library for Multilingual Unsupervised or Supervised word Embeddings [GitHub, 3246 stars]
  • ⭐ vecmap A framework to learn cross-lingual word embedding mappings [GitHub, 654 stars]
  • ⭐ sentence-transformers - Multilingual Sentence & Image Embeddings with BERT [GitHub, 18765 stars]

Social Media Oriented

  • ⭐ Ekphrasis - text processing tool, geared towards text from social networks [GitHub, 676 stars]

Phonetics

  • ⭐ DeepPhonemizer - grapheme to phoneme conversion with deep learning [GitHub, 426 stars]

Morphology

  • ⭐ LemmInflect - python module for English lemmatization and inflection [GitHub, 280 stars]
  • ⭐ Inflect - generate plurals, ordinals, indefinite articles [GitHub, 1076 stars]
  • ⭐ simplemma - simple multilingual lemmatizer for Python [GitHub, 1076 stars]

Multi-lingual tools

  • ⭐ polyglot - Multi-lingual NLP Framework [GitHub, 2367 stars]
  • ⭐ trankit - Light-Weight Transformer-based Python Toolkit for Multilingual NLP [GitHub, 795 stars]

Distributed NLP / Multi-GPU NLP

Machine Translation

  • ⭐ COMET -A Neural Framework for MT Evaluation [GitHub, 756 stars]
  • ⭐ marian-nmt - Fast Neural Machine Translation in C++ [GitHub, 1449 stars]
  • ⭐ argos-translate - Open source neural machine translation in Python [GitHub, 6092 stars]
  • ⭐ Opus-MT - Open neural machine translation models and web services [GitHub, 819 stars]
  • ⭐ dl-translate - A deep learning-based translation library built on Huggingface transformers [GitHub, 497 stars]
  • ⭐ CTranslate2 - CTranslate2 end-to-end machine translation [GitHub, 4507 stars]

Entity and String Matching

  • ⭐ PolyFuzz - Fuzzy string matching, grouping, and evaluation [GitHub, 797 stars]
  • ⭐ pyahocorasick - Python module implementing Aho-Corasick algorithm for string matching [GitHub, 1104 stars]
  • ⭐ fuzzywuzzy - Fuzzy String Matching in Python [GitHub, 9259 stars]
  • ⭐ jellyfish - approximate and phonetic matching of strings [GitHub, 2215 stars]
  • ⭐ textdistance - Compute distance between sequences [GitHub, 3534 stars]
  • ⭐ DeepMatcher - Compute distance between sequences [GitHub, 617 stars]
  • ⭐ RE2 - Simple and Effective Text Matching with Richer Alignment Features [GitHub, 341 stars]
  • ⭐ Machamp - Machamp: A Generalized Entity Matching Benchmark [GitHub, 21 stars]
  • ⭐ bge-m3 - BGE-M3 hybrid retrieval + re-ranking GitHub [GitHub, 11753 stars]
  • 📙 Cosine_Similarity_Explainer - Semantic Similarity Explainer with AI

Discourse Analysis

  • ⭐ ConvoKit - Cornell Conversational Analysis Toolkit [GitHub, 635 stars]

PII scrubbing

  • ⭐ scrubadub - Clean personally identifiable information from dirty dirty text [GitHub, 425 stars]

Hastag Segmentation

  • ⭐ hashformers - automatically inserting the missing spaces between the words in a hashtag [GitHub, 77 stars]

Books Analysis / Literary Analysis / Semantic Search

  • ⭐ booknlp - a natural language processing pipeline that scales to books and other long documents (in English) [GitHub, 918 stars]
  • ⭐ bookworm - ingests novels, builds an implicit character network and a deeply analysable graph [GitHub, 76 stars]
  • ⭐ SemanticFinder - frontend-only live semantic search with transformers.js [GitHub, 327 stars]

Non-English oriented

Japanese

  • ⭐ fugashi - Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis [GitHub, 521 stars]
  • ⭐ SudachiPy - SudachiPy is a Python version of Sudachi, a Japanese morphological analyzer [GitHub, 437 stars]
  • ⭐ Konoha - easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code [GitHub, 261 stars]
  • ⭐ jProcessing - Japanese Natural Langauge Processing Libraries [GitHub, 147 stars]
  • ⭐ Ginza - Japanese NLP Library using spaCy as framework based on Universal Dependencies [GitHub, 854 stars]
  • ⭐ kuromoji - self-contained and very easy to use Japanese morphological analyzer designed for search [GitHub, 1043 stars]
  • ⭐ nagisa - Japanese tokenizer based on recurrent neural networks [GitHub, 417 stars]
  • ⭐ KyTea - Kyoto Text Analysis Toolkit for word segmentation and pronunciation estimation [GitHub, 213 stars]
  • ⭐ Jigg - Pipeline framework for easy natural language processing [GitHub, 77 stars]
  • ⭐ Juman++ - Juman++ (a Morphological Analyzer Toolkit) [GitHub, 414 stars]
  • ⭐ RakutenMA - morphological analyzer (word segmentor + PoS Tagger) for Chinese and Japanese written purely in JavaScript [GitHub, 472 stars]
  • ⭐ toiro - a comparison tool of Japanese tokenizers [GitHub, 123 stars]

Thai

  • ⭐ AttaCut - Fast and Reasonably Accurate Word Tokenizer for Thai [GitHub, 96 stars]
  • ⭐ ThaiLMCut - Word Tokenizer for Thai Language [GitHub, 17 stars]

Chinese

  • ⭐ Spacy-pkuseg - The pkuseg toolkit for multi-domain Chinese word segmentation [GitHub, 70 stars]

Ukrainian

  • ⭐ recruitment-dataset - Recruitment Dataset Preprocessing and Recommender System (Ukrainian, English)

Other

  • ⭐ textblob-de - TextBlob: Simplified Text Processing for German [GitHub, 103 stars]
  • ⭐ Kashgari Transfer Learning with focus on Chinese [GitHub, 2384 stars]
  • ⭐ Underthesea - Vietnamese NLP Toolkit [GitHub, 1737 stars]
  • ⭐ PTT5 - Pretraining and validating the T5 model on Brazilian Portuguese data [GitHub, 91 stars]

Text Data Labelling & Classification

  • ⭐ Small-Text - Active Learning for Text Classifcation in Python [GitHub, 643 stars]
  • ⭐ Doccano - open source annotation tool for machine learning practitioners [GitHub, 10659 stars]
  • ⭐ Adala - Autonomous DAta (Labeling) Agent framework [GitHub, 1593 stars]
  • ⭐ EDA - Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks [GitHub, 1652 stars]
  • 🔱 Prodigy - annotation tool powered by active learning [Paid Service]

The-NLP-Learning

Note Section keywords: learn NLP

🔙 Back to the Table of Contents

General

Courses

Books

Tutorials

The-NLP-Communities

Other-NLP-Topics

🔙 Back to the Table of Contents

Tokenization

  • ⭐ tokenizers - Fast State-of-the-Art Tokenizers optimized for Research and Production [GitHub, 10782 stars]
  • ⭐ SentencePiece - Unsupervised text tokenizer for Neural Network-based text generation [GitHub, 11870 stars]
  • ⭐ SoMaJo - A tokenizer and sentence splitter for German and English web and social media texts [GitHub, 152 stars]

Data Augmentation and Weak Supervision

Libraries and Frameworks
  • ⭐ WildNLP Text manipulation library to test NLP models [GitHub, 76 stars]
  • ⭐ NLPAug Data augmentation for NLP [GitHub, 4657 stars]
  • ⭐ SentAugment Data augmentation by retrieving similar sentences from larger datasets [GitHub, 358 stars]
  • ⭐ TextAttack - framework for adversarial attacks, data augmentation, and model training in NLP [GitHub, 3427 stars]
  • ⭐ skweak - software toolkit for weak supervision applied to NLP tasks [GitHub, 927 stars]
  • ⭐ NL-Augmenter - Collaborative Repository of Natural Language Transformations [GitHub, 787 stars]
  • ⭐ EDA - Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks [GitHub, 1652 stars]
  • ⭐ snorkel Framework to generate training data [GitHub, 5970 stars]
  • ⭐ dialoguefactory Generate continuous dialogue data in a simulated textual world [GitHub, 5 stars]
Reading Material and Tutorials

Named Entity Recognition (NER)

Relation Extraction

  • ⭐ tacred-relation TACRED: position-aware attention model for relation extraction [GitHub, 359 stars]
  • ⭐ tacrev TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [GitHub, 70 stars]
  • ⭐ tac-self-attention Relation extraction with position-aware self-attention [GitHub, 66 stars]
  • ⭐ Re-TACRED Re-TACRED: Addressing Shortcomings of the TACRED Dataset [GitHub, 55 stars]

Coreference Resolution

Sentiment Analysis

Domain Adaptation

Low Resource NLP

Spell Correction / Error Correction

  • ⭐ Gramformer - ramework for detecting, highlighting and correcting grammatical errors [GitHub, 1581 stars]
  • ⭐ NeuSpell - A Neural Spelling Correction Toolkit [GitHub, 712 stars]
  • ⭐ SymSpellPy - Python port of SymSpell [GitHub, 871 stars]
  • 📙 Speller100 by Microsoft [Blog, Feb 2021]
  • ⭐ JamSpell - spell checking library - accurate, fast, multi-language [GitHub, 662 stars]
  • ⭐ pycorrector - spell correction for Chinese [GitHub, 6452 stars]
  • ⭐ contractions - Fixes contractions such as you're to you are [GitHub, 318 stars]
  • 📙 Fine Tuning T5 for Grammar Correction by Sachin Abeywardana [Blog, Nov 2022]

PDF Parsing

  • ⭐ spacy-layout - Process PDFs, Word documents and more with spaCy [GitHub, 902 stars]
  • ⭐ bentopdf - A Privacy First PDF Toolkit [GitHub, 13552 stars]

Style Transfer for NLP

  • ⭐ Styleformer - Neural Language Style Transfer framework [GitHub, 494 stars]
  • ⭐ StylePTB - A Compositional Benchmark for Fine-grained Controllable Text Style Transfer [GitHub, 64 stars]

Automata Theory for NLP

  • ⭐ pyahocorasick - Python module implementing Aho-Corasick algorithm for string matching [GitHub, 1104 stars]

Obscene words detection

  • ⭐ LDNOOBW - List of Dirty, Naughty, Obscene, and Otherwise Bad Words [GitHub, 3376 stars]

Reddit Analysis

  • ⭐ Subreddit Analyzer - comprehensive Data and Text Mining workflow for submissions and comments from any given public subreddit [GitHub, 499 stars]

Skill Detection

  • ⭐ SkillNER - rule based NLP module to extract job skills from text [GitHub, 209 stars]

Reinforcement Learning for NLP

  • ⭐ nlp-gym - NLPGym - A toolkit to develop RL agents to solve NLP tasks [GitHub, 201 stars]

AutoML / AutoNLP

  • ⭐ AutoNLP - Faster and easier training and deployments of SOTA NLP models [GitHub, 4574 stars]
  • ⭐ TPOT - Python Automated Machine Learning tool [GitHub, 10047 stars]
  • ⭐ Auto-PyTorch - Automatic architecture search and hyperparameter optimization for PyTorch [GitHub, 2534 stars]
  • ⭐ HungaBunga - Brute-Force all sklearn models with all parameters using .fit .predict [GitHub, 0 stars]
  • ⭐ Optuna - hyperparameter optimization framework [GitHub, 14280 stars]
  • ⭐ FLAML - fast and lightweight AutoML library [GitHub, 4360 stars]
  • ⭐ Gradsflow - open-source AutoML & PyTorch Model Training Library [GitHub, 307 stars]

OCR - Optical Character Recognition

Document AI

Text Generation

Title / Headlines Generation

  • ⭐ TitleStylist Learning to Generate Headlines with Controlled Styles [GitHub, 78 stars]

NLP research reproducibility

License CC0

Attributions

Resources

  • All linked resources belong to original authors

Icons

Fonts

The Pandect Series also includes

     

awesome-list
deeplearning
natural-language-processing
naturallanguageprocessing
nlp
pandect

Similar repos

keon/awesome-nlp

:book: A curated list of resources dedicated to Natural Language Processing (NLP)

19,028

ElizaLo/NLP-Natural-Language-Processing

Projects and useful articles / links

435

fighting41love/funNLP

中英文敏感词、语言检测、中外手机/电话归属地/运营商查询、名字推断性别、手机号抽取、身份证抽取、邮箱抽取、中日文人名库、中文缩写库、拆字词典、词汇情感值、停用词、反动词表、暴恐词表、繁简体转换、英文…

83,345

changwookjun/nlp-paper

NLP Paper

592

NiuTrans/ABigSurvey

A collection of 1000+ survey papers on Natural Language Processing (NLP) and Machine Learning (ML).

2,028

Kyubyong/nlp_tasks

Natural Language Processing Tasks and References

3,008

accelerated-text/awesome-nlg

A curated list of resources dedicated to Natural Language Generation (NLG)

483

cedrickchee/awesome-transformer-nlp

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT,…

1,151

Contributors

ivan-bilan

156 commits

axiomofjoy

2 commits

anoopkunchukuttan

2 commits

Darshnadvani

1 commits

Languages

Python

100.0%