catalyst is a C# Natural Language Processing library built for speed. Inspired by spaCy's design, it brings pre-trained models, out-of-the box support for training word and document embeddings, and flexible entity recognition models.

⚡ Features

• Fast, modern pure-C# NLP library, targeting .NET 9 and .NET 10
• Cross-platform, runs anywhere .NET core is supported - Windows, Linux, macOS and even ARM
• Non-destructive tokenization, >99.9% RegEx-free, >1M tokens/s on a modern CPU
• Named Entity Recognition (gazeteer, rule-based & perceptron-based)
• Pre-trained models based on Universal Dependencies project
• Custom models for learning Abbreviations & Senses
• Out-of-the-box support for training FastText and StarSpace embeddings (pre-trained models coming soon)
• Part-of-speech tagging
• Language detection using FastText or cld3
• Efficient binary serialization based on MessagePack
• Pre-built models for language packages ✨
• Lemmatization ✨ (using lookup tables ported from spaCy)

⚠️ Breaking changes

ILemmatizer takes the token's value, not the token. Token is a struct, so a lemmatizer that asked for an IToken boxed one on every Lemma / LemmaAsSpan read - measured at 72 bytes per read, paid once per token a search index writes. The three members now take a ReadOnlySpan<char>:

bool IsBaseForm(ReadOnlySpan<char> value);
string GetLemma(ReadOnlySpan<char> value);
ReadOnlySpan<char> GetLemmaAsSpan(ReadOnlySpan<char> value);

A custom ILemmatizer has to be updated, and a Catalyst.Models.* package built against an older Catalyst will not load - upgrade the language packages together with Catalyst. English.Map.ToAmerican / ToBritish narrow the same way (a string still works, it converts implicitly).

netstandard2.1, netcoreapp3.1 and net5.0 - net8.0 are no longer built. Catalyst and the language packages target net9.0 and net10.0.

Language Packages ✨

All language-specific data and models are provided as NuGet packages, you can find all packages here.

The new models are trained on the latest release of Universal Dependencies v2.7.

We've also added the option to store and load models using streams:

// Creates and stores the model
var isApattern = new PatternSpotter(Language.English, 0, tag: "is-a-pattern", captureTag: "IsA");
isApattern.NewPattern(
    "Is+Noun",
    mp => mp.Add(
        new PatternUnit(P.Single().WithToken("is").WithPOS(PartOfSpeech.VERB)),
        new PatternUnit(P.Multiple().WithPOS(PartOfSpeech.NOUN, PartOfSpeech.PROPN, PartOfSpeech.AUX, PartOfSpeech.DET, PartOfSpeech.ADJ))
));
using(var f = File.OpenWrite("my-pattern-spotter.bin"))
{
    await isApattern.StoreAsync(f);
}

// Load the model back from disk
var isApattern2 = new PatternSpotter(Language.English, 0, tag: "is-a-pattern", captureTag: "IsA");

using(var f = File.OpenRead("my-pattern-spotter.bin"))
{
    await isApattern2.LoadAsync(f);
}

✨ Getting Started

Using catalyst is as simple as installing its NuGet Package, and setting the storage to use our online repository. This way, models will be lazy loaded either from disk or downloaded from our online repository. Check out also some of the sample projects for more examples on how to use catalyst.

Catalyst.Models.English.Register(); //You need to pre-register each language (and install the respective NuGet Packages)

Storage.Current = new DiskStorage("catalyst-models");
var nlp = await Pipeline.ForAsync(Language.English);
var doc = new Document("The quick brown fox jumps over the lazy dog", Language.English);
nlp.ProcessSingle(doc);
Console.WriteLine(doc.ToJson());

You can also take advantage of C# lazy evaluation and native multi-threading support to process a large number of documents in parallel:

var docs = GetDocuments();
var parsed = nlp.Process(docs);
DoSomething(parsed);

IEnumerable<IDocument> GetDocuments()
{
    //Generates a few documents, to demonstrate multi-threading & lazy evaluation
    for(int i = 0; i < 1000; i++)
    {
        yield return new Document("The quick brown fox jumps over the lazy dog", Language.English);
    }
}

void DoSomething(IEnumerable<IDocument> docs)
{
    foreach(var doc in docs)
    {
        Console.WriteLine(doc.ToJson());
    }
}

Training a new FastText word2vec embedding model is as simple as this:

var nlp = await Pipeline.ForAsync(Language.English);
var ft = new FastText(Language.English, 0, "wiki-word2vec");
ft.Data.Type = FastText.ModelType.CBow;
ft.Data.Loss = FastText.LossType.NegativeSampling;
ft.Train(nlp.Process(GetDocs()));
ft.StoreAsync();

For fast embedding search, we have also released a C# version of the "Hierarchical Navigable Small World" (HNSW) algorithm on NuGet, based on our fork of Microsoft's HNSW.Net. We have also released a C# version of the "Uniform Manifold Approximation and Projection" (UMAP) algorithm for dimensionality reduction on GitHub and on NuGet.

📖 Links