✂️ Sentence segmentation with wtpsplit's state-of-the-art Segment any Text (SaT) models
Python
43
8 commits
updated May 2, 2026
🪓 wtpsplit is a Python package that offers training, inference, and evaluation of state-of-the-art Segment any Text (SaT) models for partitioning text into sentences.
✂️ wtpsplit-lite is a lightweight version of wtsplit that only retains accelerated ONNX inference of SaT models with minimal dependencies:
To install this package, run:
pip install wtpsplit-lite
[!TIP] For a complete list of Segment any Text (SaT) models and all
SaT.splitkeyword arguments, see the wtsplit README.
Example usage:
from wtpsplit_lite import SaT
text = """
It is known that Maxwell’s electrodynamics—as usually understood at the
present time—when applied to moving bodies, leads to asymmetries which do
not appear to be inherent in the phenomena. Take, for example, the recipro-
cal electrodynamic action of a magnet and a conductor.
"""
# Fast (~150ms/page), good quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256)
# Slow, highest quality:
sat = SaT("sat-12l-sm")
sentences = sat.split(text)
This package also contributes a new 'hat' weighting scheme to wtpsplit that improves output quality when using large strides. To enable it, set weighting="hat" as follows:
# Fast (~150ms/page), better quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256, weighting="hat")
[!NOTE] In wtpsplit, the SaT implementation treats newlines as sentence boundaries by default. However, this leads to poor results on text extracted from PDF such as in the example above. In wtpsplit-lite, newlines are therefore treated as whitepace by default. You can choose which behavior you prefer with the
treat_newline_as_spaceboolean keyword argument of theSaT.splitmethod.
Generate an SSH key and add the SSH key to your GitHub account.
Configure SSH to automatically load your SSH keys:
cat << EOF >> ~/.ssh/config
Host *
AddKeysToAgent yes
IgnoreUnknown UseKeychain
UseKeychain yes
ForwardAgent yes
EOF
Install VS Code and VS Code's Dev Containers extension. Alternatively, install PyCharm.
Optional: install a Nerd Font such as FiraCode Nerd Font and configure VS Code or PyCharm to use it.
The following development environments are supported:
⭐️ GitHub Codespaces: click on Open in GitHub Codespaces to start developing in your browser.
⭐️ VS Code Dev Container (with container volume): click on Open in Dev Containers to clone this repository in a container volume and create a Dev Container with VS Code.
⭐️ uv: clone this repository and run the following from root of the repository:
# Create and install a virtual environment
uv sync --python 3.10 --all-extras
# Activate the virtual environment
source .venv/bin/activate
# Install the pre-commit hooks
pre-commit install --install-hooks
VS Code Dev Container: clone this repository, open it with VS Code, and run Ctrl/⌘ + ⇧ + P → Dev Containers: Reopen in Container.
PyCharm Dev Container: clone this repository, open it with PyCharm, create a Dev Container with Mount Sources, and configure an existing Python interpreter at /opt/venv/bin/python.
poe from within the development environment to print a list of Poe the Poet tasks available to run on this project.uv add {package} from within the development environment to install a run time dependency and add it to pyproject.toml and uv.lock. Add --dev to install a development dependency.uv sync --upgrade from within the development environment to upgrade all dependencies to the latest versions allowed by pyproject.toml. Add --only-dev to upgrade the development dependencies only.cz bump to bump the package's version, update the CHANGELOG.md, and create a git tag. Then push the changes and the git tag with git push origin main --tags.✂️ Sentence segmentation with wtpsplit's state-of-the-art Segment any Text (SaT) models
Python
43
8 commits
updated May 2, 2026
🪓 wtpsplit is a Python package that offers training, inference, and evaluation of state-of-the-art Segment any Text (SaT) models for partitioning text into sentences.
✂️ wtpsplit-lite is a lightweight version of wtsplit that only retains accelerated ONNX inference of SaT models with minimal dependencies:
To install this package, run:
pip install wtpsplit-lite
[!TIP] For a complete list of Segment any Text (SaT) models and all
SaT.splitkeyword arguments, see the wtsplit README.
Example usage:
from wtpsplit_lite import SaT
text = """
It is known that Maxwell’s electrodynamics—as usually understood at the
present time—when applied to moving bodies, leads to asymmetries which do
not appear to be inherent in the phenomena. Take, for example, the recipro-
cal electrodynamic action of a magnet and a conductor.
"""
# Fast (~150ms/page), good quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256)
# Slow, highest quality:
sat = SaT("sat-12l-sm")
sentences = sat.split(text)
This package also contributes a new 'hat' weighting scheme to wtpsplit that improves output quality when using large strides. To enable it, set weighting="hat" as follows:
# Fast (~150ms/page), better quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256, weighting="hat")
[!NOTE] In wtpsplit, the SaT implementation treats newlines as sentence boundaries by default. However, this leads to poor results on text extracted from PDF such as in the example above. In wtpsplit-lite, newlines are therefore treated as whitepace by default. You can choose which behavior you prefer with the
treat_newline_as_spaceboolean keyword argument of theSaT.splitmethod.
Generate an SSH key and add the SSH key to your GitHub account.
Configure SSH to automatically load your SSH keys:
cat << EOF >> ~/.ssh/config
Host *
AddKeysToAgent yes
IgnoreUnknown UseKeychain
UseKeychain yes
ForwardAgent yes
EOF
Install VS Code and VS Code's Dev Containers extension. Alternatively, install PyCharm.
Optional: install a Nerd Font such as FiraCode Nerd Font and configure VS Code or PyCharm to use it.
The following development environments are supported:
⭐️ GitHub Codespaces: click on Open in GitHub Codespaces to start developing in your browser.
⭐️ VS Code Dev Container (with container volume): click on Open in Dev Containers to clone this repository in a container volume and create a Dev Container with VS Code.
⭐️ uv: clone this repository and run the following from root of the repository:
# Create and install a virtual environment
uv sync --python 3.10 --all-extras
# Activate the virtual environment
source .venv/bin/activate
# Install the pre-commit hooks
pre-commit install --install-hooks
VS Code Dev Container: clone this repository, open it with VS Code, and run Ctrl/⌘ + ⇧ + P → Dev Containers: Reopen in Container.
PyCharm Dev Container: clone this repository, open it with PyCharm, create a Dev Container with Mount Sources, and configure an existing Python interpreter at /opt/venv/bin/python.
poe from within the development environment to print a list of Poe the Poet tasks available to run on this project.uv add {package} from within the development environment to install a run time dependency and add it to pyproject.toml and uv.lock. Add --dev to install a development dependency.uv sync --upgrade from within the development environment to upgrade all dependencies to the latest versions allowed by pyproject.toml. Add --only-dev to upgrade the development dependencies only.cz bump to bump the package's version, update the CHANGELOG.md, and create a git tag. Then push the changes and the git tag with git push origin main --tags.14 followers · starred Jan 2026