The documentation and code is available on Github alasdairforsythe/tokenmonster.
The pretrained vocabularies are all available for download here.
July 11: TokenMonster v1.1.1 has been released. The "420" prebuilt vocabularies are being released as they are completed, at a rate of around 10 per day.
Choose a dataset from:
codeenglishenglishcodefictionChoose a vocab size from:
1024204840968000160002400032000400005025665536100256Choose an optimization mode from:
unfilteredcleanbalancedconsistentstrictFor a capcode disabled vocabulary add:
nocapcodeAnd finally add the version number:
v1Examples:
fiction-24000-consistent-v1code-4096-clean-nocapcode-v1There are two additional vocabularies:
gpt2llamaThe documentation and code is available on Github alasdairforsythe/tokenmonster.
The pretrained vocabularies are all available for download here.
July 11: TokenMonster v1.1.1 has been released. The "420" prebuilt vocabularies are being released as they are completed, at a rate of around 10 per day.
Choose a dataset from:
codeenglishenglishcodefictionChoose a vocab size from:
1024204840968000160002400032000400005025665536100256Choose an optimization mode from:
unfilteredcleanbalancedconsistentstrictFor a capcode disabled vocabulary add:
nocapcodeAnd finally add the version number:
v1Examples:
fiction-24000-consistent-v1code-4096-clean-nocapcode-v1There are two additional vocabularies:
gpt2llama