A Japanese NLP Library using spaCy as framework based on Universal Dependencies
Python
873
634 commits
updated Sep 30, 2026
An Open Source Japanese NLP Library, based on Universal Dependencies
Please read the Breaking Changes before you upgrade GiNZA.
The GiNZA parsing models are released as part of the results of a joint research project between Recruit Co., Ltd. and the National Institute for Japanese Language and Linguistics.
Please also read the English documentation.
From GiNZA v5.3.0, the runtime environment has changed to Python 3.10 or later. We recommend using Python 3.12 because of simpler dependency installation process and higher throughput. We do not recommend to use Anaconda environment because the pip install step may not work properly.
Please also see the Development Environment section below.
[!NOTE] The GiNZA dependency parsing can be significantly accelerated with a GPU. The hardware acceleration is enabled by default in Mac OS environments running on Apple Silicon. For details, see 3. Enabling the GPU。
Running the Transformers model requires at least 16GB of memory. If you have insufficient memory, please try the standard model described below.
Install the Transformers model (ja_ginza_electra or ja_ginza_bert_large) by running one of the following commands:
(This will also install GiNZA and related Transformers libraries.)
$ pip install -U ja_ginza_electra
$ pip install -U ja_ginza_bert_large
[!NOTE] The model package installed by the above command does not include large transformer models or tokenizers. These large files are automatically downloaded from Hugging Face Hub on the first run, and the locally cached files are used for subsequent runs. The first time you run transformers model, it will take several seconds to tens of seconds to start up due to the initialization process.
Run the following command to install the standard model ja_ginza.
(GiNZA-related libraries will also be installed at the same time.)
$ pip install -U ja_ginza
On Mac OS environments running on Apple Silicon, the hardware acceleration is automatically enabled under the following conditions:
ja_ginza
thinc-apple-opsthinc-apple-ops is not supported in 3.13 and later)thinc-apple-ops on MacBook Pro M4 Max 128GB is 2.6 times for Python 3.10-3.11 and 5.2 times for Python 3.12 in ja_ginza.ja_ginza_electra and ja_ginza_large_bert
torch.backends.mpstorch.backends.mps on MacBook Pro M4 Max 128GB is 1.3 times for Python 3.10-13 in ja_ginza_bert_large.To enable NVIDIA GPU acceleration in a Linux OS environment, install CUDA on Linux, add the path to the CUDA libraries to the environment variable LD_LIBRARY_PATH like export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH, and then install the ginza package specifying the CUDA version in extras as follows:
$ pip install ginza[cuda11x]$ pip install ginza[cuda12x]$ pip install ginza[cuda13x]The ginza command enables GPU acceleration by default if a GPU is available.
The -g option of the ginza command allows you to specify the device number of the GPU to use (specifying -1 disables GPU acceleration).
$ ginza -g 0
When running the ginza command without the -g option, the following init log is output if GPU acceleration is enabled:
$ ginza
GPU #0 enabled
When executing standard model in Mac OS running on Apple Silicon, if hardware acceleration is enabled, the init log will be:
$ ginza
thinc-apple-ops enabled
Run ginza command from the console, then input some Japanese text.
After pressing enter key, you will get the parsed results with CoNLL-U Syntactic Annotation format.
$ ginza
銀座でランチをご一緒しましょう。
# text = 銀座でランチをご一緒しましょう。
1 銀座 銀座 PROPN 名詞-固有名詞-地名-一般 _ 6 nmod _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|NP_B|Reading=ギンザ|NE=B-GPE|ENE=B-City|ClauseHead=6
2 で で ADP 助詞-格助詞 _ 1 case _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Reading=デ|ClauseHead=6
3 ランチ ランチ NOUN 名詞-普通名詞-一般 _ 6 obj _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|NP_B|Reading=ランチ|ClauseHead=6
4 を を ADP 助詞-格助詞 _ 3 case _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Reading=ヲ|ClauseHead=6
5 ご ご NOUN 接頭辞 _ 6 compound _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=CONT|NP_B|Reading=ゴ|ClauseHead=6
6 一緒 一緒 NOUN 名詞-普通名詞-サ変可能 _ 0 root _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=ROOT|NP_I|Reading=イッショ|ClauseHead=6
7 し する AUX 動詞-非自立可能 _ 6 aux _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Inf=サ行変格,連用形-一般|Reading=シ|ClauseHead=6
8 ましょう ます AUX 助動詞 _ 6 aux _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Inf=助動詞-マス,意志推量形|Reading=マショウ|ClauseHead=6
9 。 。 PUNCT 補助記号-句点 _ 6 punct _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=CONT|Reading=。|ClauseHead=6
ginzame command provides tokenization function like MeCab.
The output format of ginzame is almost same as mecab, but the last pronunciation field is always '*'.
$ ginzame
銀座でランチをご一緒しましょう。
銀座 名詞,固有名詞,地名,一般,*,*,銀座,ギンザ,*
で 助詞,格助詞,*,*,*,*,で,デ,*
ランチ 名詞,普通名詞,一般,*,*,*,ランチ,ランチ,*
を 助詞,格助詞,*,*,*,*,を,ヲ,*
ご 接頭辞,*,*,*,*,*,御,ゴ,*
一緒 名詞,普通名詞,サ変可能,*,*,*,一緒,イッショ,*
し 動詞,非自立可能,*,*,サ行変格,連用形-一般,為る,シ,*
ましょう 助動詞,*,*,*,助動詞-マス,意志推量形,ます,マショウ,*
。 補助記号,句点,*,*,*,*,。,。,*
EOS
The format of spaCy's JSON is available by specifying -f 3 or -f json for ginza command.
$ ginza -f json
銀座でランチをご一緒しましょう。
[
{
"paragraphs": [
{
"raw": "銀座でランチをご一緒しましょう。",
"sentences": [
{
"tokens": [
{"id": 1, "orth": "銀座", "tag": "名詞-固有名詞-地名-一般", "pos": "PROPN", "lemma": "銀座", "head": 5, "dep": "obl", "ner": "B-City"},
{"id": 2, "orth": "で", "tag": "助詞-格助詞", "pos": "ADP", "lemma": "で", "head": -1, "dep": "case", "ner": "O"},
{"id": 3, "orth": "ランチ", "tag": "名詞-普通名詞-一般", "pos": "NOUN", "lemma": "ランチ", "head": 3, "dep": "obj", "ner": "O"},
{"id": 4, "orth": "を", "tag": "助詞-格助詞", "pos": "ADP", "lemma": "を", "head": -1, "dep": "case", "ner": "O"},
{"id": 5, "orth": "ご", "tag": "接頭辞", "pos": "NOUN", "lemma": "ご", "head": 1, "dep": "compound", "ner": "O"},
{"id": 6, "orth": "一緒", "tag": "名詞-普通名詞-サ変可能", "pos": "VERB", "lemma": "一緒", "head": 0, "dep": "ROOT", "ner": "O"},
{"id": 7, "orth": "し", "tag": "動詞-非自立可能", "pos": "AUX", "lemma": "する", "head": -1, "dep": "advcl", "ner": "O"},
{"id": 8, "orth": "ましょう", "tag": "助動詞", "pos": "AUX", "lemma": "ます", "head": -2, "dep": "aux", "ner": "O"},
{"id": 9, "orth": "。", "tag": "補助記号-句点", "pos": "PUNCT", "lemma": "。", "head": -3, "dep": "punct", "ner": "O"}
]
}
]
}
]
}
]
If you want to use cabocha -f1 (lattice style) like output, add -f 1 or -f cabocha option to ginza command.
This option's format is almost same as cabocha -f1 but the func_index field (after the slash) is slightly different.
Our func_index field indicates the boundary where the 自立語 ends in each 文節 (and the 機能語 might start from there).
And the functional token filter is also slightly different between cabocha -f1 and ' ginza -f cabocha.
$ ginza -f cabocha
銀座でランチをご一緒しましょう。
* 0 2D 0/1 0.000000
銀座 名詞,固有名詞,地名,一般,,銀座,ギンザ,* B-City
で 助詞,格助詞,*,*,,で,デ,* O
* 1 2D 0/1 0.000000
ランチ 名詞,普通名詞,一般,*,,ランチ,ランチ,* O
を 助詞,格助詞,*,*,,を,ヲ,* O
* 2 -1D 0/2 0.000000
ご 接頭辞,*,*,*,,ご,ゴ,* O
一緒 名詞,普通名詞,サ変可能,*,,一緒,イッショ,* O
し 動詞,非自立可能,*,*,サ行変格,連用形-一般,する,シ,* O
ましょう 助動詞,*,*,*,助動詞-マス,意志推量形,ます,マショウ,* O
。 補助記号,句点,*,*,,。,。,* O
EOS
We added -p NUM_PROCESS option from GiNZA v3.0.
Please specify the number of analyzing processes to NUM_PROCESS.
You might want to use all the cpu cores for GiNZA, then execute ginza -p 0.
The memory requirement is about 130MB/process (to be improved).
Following steps shows dependency parsing results with sentence boundary 'EOS'.
import spacy
nlp = spacy.load('ja_ginza_electra')
doc = nlp('銀座でランチをご一緒しましょう。')
for sent in doc.sents:
for token in sent:
print(
token.i,
token.orth_,
token.lemma_,
token.norm_,
token.morph.get("Reading"),
token.pos_,
token.morph.get("Inflection"),
token.tag_,
token.dep_,
token.head.i,
)
print('EOS')
The user dictionary files should be set to userDict field of sudachi.json in the installed package directory ofja_ginza_dict package.
Please read the official documents to compile user dictionaries with sudachipy command.
SudachiPy - User defined Dictionary
Sudachi User Dictionary Construction (Japanese Only)
GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models are distributed under the MIT License. You must agree and follow the MIT License to use GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models.
spaCy is the key framework of GiNZA.
SudachiPy provides high accuracies for tokenization and pos tagging.
Sudachi LICENSE PAGE, SudachiPy LICENSE PAGE, SudachiDict LEGAL PAGE, chiVe LICENSE PAGE
The GiNZA v5 transformer models (ja_ginza_electra and ja_ginza_bert_large) use Hugging Face Transformers as inference framework.
The parsing model of GiNZA v5 is trained on a part of UD Japanese BCCWJ r2.8 (Omura and Asahara:2018).
@inproceedings{omura-asahara-2018-ud,
title = "{UD}-{J}apanese {BCCWJ}: {U}niversal {D}ependencies Annotation for the {B}alanced {C}orpus of {C}ontemporary {W}ritten {J}apanese",
author = "Omura, Mai and
Asahara, Masayuki",
booktitle = "Proceedings of the Second Workshop on Universal Dependencies ({UDW} 2018)",
month = nov,
year = "2018",
address = "Brussels, Belgium",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/W18-6014/",
doi = "10.18653/v1/W18-6014",
pages = "117--125"
}
The named entity recognition model of GiNZA v5 is trained on a part of GSK2014-A (2019) BCCWJ edition (Hashimoto, Inui, and Murakami:2008). We use two of the named entity label systems, both Sekine's Extended Named Entity Hierarchy and extended OntoNotes5. This model is developed by National Institute for Japanese Language and Linguistics, and Megagon Labs.
ja_ginza_electra is a fine-tuned model of transformers-ud-japanese-electra-base-discriminator which is pretrained on more than 200 million Japanese sentences extracted from mC4.
The mC4 is published under the ODC Attribution License.
@article{2019t5,
author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
journal = {arXiv e-prints},
year = {2019},
archivePrefix = {arXiv},
eprint = {1910.10683},
}
ja_ginza_bert_large, ja_ginza_electra, and ja_ginza.ginza command enables GPU acceleration by default under certain conditions.
ginza -g -1.thinc-apple-ops on Mac OS running Apple Silicon are 3.10 through 3.12.ja_ginza_bert_largeginza-transformers was upgraded to v1.4.0.
spacy-transformers has a requirement of torch>=1.8.0 and transformers<4.53.3.ja_ginza_bert_largedoc.user_data["reading_forms"][token.i] -> token.morph.get("Reading")doc.user_data["inflections"][token.i] -> token.morph.get("Inflection")force_using_normalized_form_as_lemma(True) -> token.norm_ginza -m en_core_web_mdginza --require_gpu and ginza -g to take a gpu_id argument
gpu_id value is -1 which uses only CPUsginza -f json option always analyze the line which starts with # regardless the option value of -c. #215ginza -p {n_process} and ginzame) of ginza command #204Bunsetu span should not cross the sentence boundary #195Command Line -s option and set_split_mode() not working in v5.0.x #185ginzame not woriking in ginza ver. 5 #179Command Line -d option not working in v5.0.0 #178ja-ginza and ja-ginza-electra for -m option of ginza commandja-ginza-electra modelja-ginza model by adding morphologizer to the tail of spaCy pipelineginza package
pip install -U ginza ja-ginza-electrapip install -U ginza ja-ginzaCompoundSplitter and BunsetuRecognizer to compound_splitter and bunsetu_recognizer respectively-n
normalized_form as Token.lemma_-m (ja_ginza|ja_ginza_electra)
Degital_Game to Digital_Game-d option, which disables spaCy's sentence separator, to ginza command line toolginza command line tool works correctly without BunsetuRecognizer in the pipelineCompoundSplitter for nlp.to_disk()spacy.lang.ja of spaCy v2.3
Token.lemma_ with the output of SudachiPy's Morpheme.dictionary_form()ja_ginza_dict package safetyB-GPEToken._.* and add some entries for Doc.user_data[] and accessors
ginza.inflection(Token))ginza.reading_form(Token))ginza.bunsetu_bi_label(Token))ginza.bunsetu_position_type(Token))ginza.is_bunsetu_head(Token))from ginza import *)
Token._.sudachi field would be set after calling SudachipyTokenizer.set_enable_ex_sudachi(True), to avoid serializtion errorsimport spacy
import pickle
nlp = spacy.load('ja_ginza')
doc1 = nlp('This example will be serialized correctly.')
doc1.to_bytes()
with open('sample1.pickle', 'wb') as f:
pickle.dump(doc1, f)
nlp.tokenizer.set_enable_ex_sudachi(True)
doc2 = nlp('This example will cause a serialization error.')
doc2.to_bytes()
with open('sample2.pickle', 'wb') as f:
pickle.dump(doc2, f)
ja_ginza_dict from PyPIginza and ginzame
-i option to initialize the files of ja_ginza_dictginza and ja_ginza from PyPI
pip install ginza, and run ginzaja_ginza, is also available from PyPI.token.ent_type_ value is changed to Sekine's Extended Named Entity Hierarchy
ENE7 attribute to the last field of the output of ginzatoken._.ne
PHONE, EMAIL, URL, and PET_NAMEspacy pretrain over 100 epochs
spacy train effectively working on UD Japanese BCCWJSudachiDict_core-20191224ginzame
sudachipy by multiprocessing.Pool and output results with mecab like formatsudachipy command requires additional SudachiDict package installationginza (ginza.command_line.main_ginza)
mode to sudachipy_modedisable_pipes and recreate_correctorhash_comment, parallel, filesmecab to the choices for the argument of -f optionparallel NUM_PROCESS option (EXPERIMENTAL)ENE7 attribute to conllu miscellaneous field
ginza.ent_type_mapping.ENE_NE_MAPPING is used to convert ENE7 label to NEginzame (ginza.command_line.main_ginzame)
mecab like output formattoken._.ne for ner labelginza/sudachipy_tokenizer.py
SudachiTokenizer to SudachipyTokenizerSUDACHI_DEFAULT_SPLIT_MODE instead of SUDACHI_DEFAULT_SPLITMODE or SUDACHI_DEFAULT_MODEspacy to v2.2.3sudachipy to v0.4.2as_* type dependencies completelysplit_mode has been set incorrectly to sudachipy.tokenizer from v2.0.0 (#43)
split_mode incompatibility between the training phase and the ginza command.split_mode was set to 'B' for training phase and python APIs, but 'C' for ginza command.split_mode to 'C' entirely.-f and --output-format option to ginza command:
-f 0 or -f conllu : CoNLL-U Syntactic Annotation format-f 1 or -f cabocha: cabocha -f1 compatible formatbunsetu_index : bunsetu index starting from 0reading: reading of token (not a pronunciation)sudachi: SudachiPy's morpheme instance (or its list when then tokens are gathered by JapaneseCorrector)spacy pretrain command to capture the language model from UD-Japanese BCCWJ, UD_Japanese-PUD and KWDLC.-pt 'tag,dep' option of spacy trainginza command
ginza from the consoleginzaja_ginzaspacy.lang.ja is overridden by ginzasudachipy related directories
pip during ginza installationtoken._.bunsetu_bi_label, token._.bunsetu_position_typetoken._.infpos_detail (same value is set to token.tag_)$ git clone 'https://github.com/megagonlabs/ginza.git'
For normal environment:
$ python setup.py develop
Copy system.dic from installed package directory of ja_ginza_dict to ./ja_ginza_dict/sudachidict/.
The analysis model of GiNZA is trained by spacy train command.
$ python -m spacy train ja ja_ginza-4.0.0 corpus/ja_ginza-ud-train.json corpus/ja_ginza-ud-dev.json -b ja_vectors_chive_mc90_35k/ -ovl 0.3 -n 100 -m meta.json.ginza -V 4.0.0
GiNZA uses the pytest framework for testing, and you can run the tests via setup.py without install test requirements explicitly.
Some tests depends on the ginza parsing models, so install them before the tests is needed.
$ pip install ja-ginza ja-ginza-electra
$ pip install -e .
# full test
$ python setup.py test
# test single file
$ python setup.py test --addopts ginza/tests/test_analyzer.py
Overall
Maintained
Code review
PyPI
Latest release
Releases this year
2,848 followers · starred Jul 2021
156 followers · starred May 2020
228 followers · starred May 2024
859 followers · starred Jul 2020
A Japanese NLP Library using spaCy as framework based on Universal Dependencies
Python
873
634 commits
updated Sep 30, 2026
An Open Source Japanese NLP Library, based on Universal Dependencies
Please read the Breaking Changes before you upgrade GiNZA.
The GiNZA parsing models are released as part of the results of a joint research project between Recruit Co., Ltd. and the National Institute for Japanese Language and Linguistics.
Please also read the English documentation.
From GiNZA v5.3.0, the runtime environment has changed to Python 3.10 or later. We recommend using Python 3.12 because of simpler dependency installation process and higher throughput. We do not recommend to use Anaconda environment because the pip install step may not work properly.
Please also see the Development Environment section below.
[!NOTE] The GiNZA dependency parsing can be significantly accelerated with a GPU. The hardware acceleration is enabled by default in Mac OS environments running on Apple Silicon. For details, see 3. Enabling the GPU。
Running the Transformers model requires at least 16GB of memory. If you have insufficient memory, please try the standard model described below.
Install the Transformers model (ja_ginza_electra or ja_ginza_bert_large) by running one of the following commands:
(This will also install GiNZA and related Transformers libraries.)
$ pip install -U ja_ginza_electra
$ pip install -U ja_ginza_bert_large
[!NOTE] The model package installed by the above command does not include large transformer models or tokenizers. These large files are automatically downloaded from Hugging Face Hub on the first run, and the locally cached files are used for subsequent runs. The first time you run transformers model, it will take several seconds to tens of seconds to start up due to the initialization process.
Run the following command to install the standard model ja_ginza.
(GiNZA-related libraries will also be installed at the same time.)
$ pip install -U ja_ginza
On Mac OS environments running on Apple Silicon, the hardware acceleration is automatically enabled under the following conditions:
ja_ginza
thinc-apple-opsthinc-apple-ops is not supported in 3.13 and later)thinc-apple-ops on MacBook Pro M4 Max 128GB is 2.6 times for Python 3.10-3.11 and 5.2 times for Python 3.12 in ja_ginza.ja_ginza_electra and ja_ginza_large_bert
torch.backends.mpstorch.backends.mps on MacBook Pro M4 Max 128GB is 1.3 times for Python 3.10-13 in ja_ginza_bert_large.To enable NVIDIA GPU acceleration in a Linux OS environment, install CUDA on Linux, add the path to the CUDA libraries to the environment variable LD_LIBRARY_PATH like export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH, and then install the ginza package specifying the CUDA version in extras as follows:
$ pip install ginza[cuda11x]$ pip install ginza[cuda12x]$ pip install ginza[cuda13x]The ginza command enables GPU acceleration by default if a GPU is available.
The -g option of the ginza command allows you to specify the device number of the GPU to use (specifying -1 disables GPU acceleration).
$ ginza -g 0
When running the ginza command without the -g option, the following init log is output if GPU acceleration is enabled:
$ ginza
GPU #0 enabled
When executing standard model in Mac OS running on Apple Silicon, if hardware acceleration is enabled, the init log will be:
$ ginza
thinc-apple-ops enabled
Run ginza command from the console, then input some Japanese text.
After pressing enter key, you will get the parsed results with CoNLL-U Syntactic Annotation format.
$ ginza
銀座でランチをご一緒しましょう。
# text = 銀座でランチをご一緒しましょう。
1 銀座 銀座 PROPN 名詞-固有名詞-地名-一般 _ 6 nmod _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|NP_B|Reading=ギンザ|NE=B-GPE|ENE=B-City|ClauseHead=6
2 で で ADP 助詞-格助詞 _ 1 case _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Reading=デ|ClauseHead=6
3 ランチ ランチ NOUN 名詞-普通名詞-一般 _ 6 obj _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|NP_B|Reading=ランチ|ClauseHead=6
4 を を ADP 助詞-格助詞 _ 3 case _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Reading=ヲ|ClauseHead=6
5 ご ご NOUN 接頭辞 _ 6 compound _ SpaceAfter=No|BunsetuBILabel=B|BunsetuPositionType=CONT|NP_B|Reading=ゴ|ClauseHead=6
6 一緒 一緒 NOUN 名詞-普通名詞-サ変可能 _ 0 root _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=ROOT|NP_I|Reading=イッショ|ClauseHead=6
7 し する AUX 動詞-非自立可能 _ 6 aux _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Inf=サ行変格,連用形-一般|Reading=シ|ClauseHead=6
8 ましょう ます AUX 助動詞 _ 6 aux _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|Inf=助動詞-マス,意志推量形|Reading=マショウ|ClauseHead=6
9 。 。 PUNCT 補助記号-句点 _ 6 punct _ SpaceAfter=No|BunsetuBILabel=I|BunsetuPositionType=CONT|Reading=。|ClauseHead=6
ginzame command provides tokenization function like MeCab.
The output format of ginzame is almost same as mecab, but the last pronunciation field is always '*'.
$ ginzame
銀座でランチをご一緒しましょう。
銀座 名詞,固有名詞,地名,一般,*,*,銀座,ギンザ,*
で 助詞,格助詞,*,*,*,*,で,デ,*
ランチ 名詞,普通名詞,一般,*,*,*,ランチ,ランチ,*
を 助詞,格助詞,*,*,*,*,を,ヲ,*
ご 接頭辞,*,*,*,*,*,御,ゴ,*
一緒 名詞,普通名詞,サ変可能,*,*,*,一緒,イッショ,*
し 動詞,非自立可能,*,*,サ行変格,連用形-一般,為る,シ,*
ましょう 助動詞,*,*,*,助動詞-マス,意志推量形,ます,マショウ,*
。 補助記号,句点,*,*,*,*,。,。,*
EOS
The format of spaCy's JSON is available by specifying -f 3 or -f json for ginza command.
$ ginza -f json
銀座でランチをご一緒しましょう。
[
{
"paragraphs": [
{
"raw": "銀座でランチをご一緒しましょう。",
"sentences": [
{
"tokens": [
{"id": 1, "orth": "銀座", "tag": "名詞-固有名詞-地名-一般", "pos": "PROPN", "lemma": "銀座", "head": 5, "dep": "obl", "ner": "B-City"},
{"id": 2, "orth": "で", "tag": "助詞-格助詞", "pos": "ADP", "lemma": "で", "head": -1, "dep": "case", "ner": "O"},
{"id": 3, "orth": "ランチ", "tag": "名詞-普通名詞-一般", "pos": "NOUN", "lemma": "ランチ", "head": 3, "dep": "obj", "ner": "O"},
{"id": 4, "orth": "を", "tag": "助詞-格助詞", "pos": "ADP", "lemma": "を", "head": -1, "dep": "case", "ner": "O"},
{"id": 5, "orth": "ご", "tag": "接頭辞", "pos": "NOUN", "lemma": "ご", "head": 1, "dep": "compound", "ner": "O"},
{"id": 6, "orth": "一緒", "tag": "名詞-普通名詞-サ変可能", "pos": "VERB", "lemma": "一緒", "head": 0, "dep": "ROOT", "ner": "O"},
{"id": 7, "orth": "し", "tag": "動詞-非自立可能", "pos": "AUX", "lemma": "する", "head": -1, "dep": "advcl", "ner": "O"},
{"id": 8, "orth": "ましょう", "tag": "助動詞", "pos": "AUX", "lemma": "ます", "head": -2, "dep": "aux", "ner": "O"},
{"id": 9, "orth": "。", "tag": "補助記号-句点", "pos": "PUNCT", "lemma": "。", "head": -3, "dep": "punct", "ner": "O"}
]
}
]
}
]
}
]
If you want to use cabocha -f1 (lattice style) like output, add -f 1 or -f cabocha option to ginza command.
This option's format is almost same as cabocha -f1 but the func_index field (after the slash) is slightly different.
Our func_index field indicates the boundary where the 自立語 ends in each 文節 (and the 機能語 might start from there).
And the functional token filter is also slightly different between cabocha -f1 and ' ginza -f cabocha.
$ ginza -f cabocha
銀座でランチをご一緒しましょう。
* 0 2D 0/1 0.000000
銀座 名詞,固有名詞,地名,一般,,銀座,ギンザ,* B-City
で 助詞,格助詞,*,*,,で,デ,* O
* 1 2D 0/1 0.000000
ランチ 名詞,普通名詞,一般,*,,ランチ,ランチ,* O
を 助詞,格助詞,*,*,,を,ヲ,* O
* 2 -1D 0/2 0.000000
ご 接頭辞,*,*,*,,ご,ゴ,* O
一緒 名詞,普通名詞,サ変可能,*,,一緒,イッショ,* O
し 動詞,非自立可能,*,*,サ行変格,連用形-一般,する,シ,* O
ましょう 助動詞,*,*,*,助動詞-マス,意志推量形,ます,マショウ,* O
。 補助記号,句点,*,*,,。,。,* O
EOS
We added -p NUM_PROCESS option from GiNZA v3.0.
Please specify the number of analyzing processes to NUM_PROCESS.
You might want to use all the cpu cores for GiNZA, then execute ginza -p 0.
The memory requirement is about 130MB/process (to be improved).
Following steps shows dependency parsing results with sentence boundary 'EOS'.
import spacy
nlp = spacy.load('ja_ginza_electra')
doc = nlp('銀座でランチをご一緒しましょう。')
for sent in doc.sents:
for token in sent:
print(
token.i,
token.orth_,
token.lemma_,
token.norm_,
token.morph.get("Reading"),
token.pos_,
token.morph.get("Inflection"),
token.tag_,
token.dep_,
token.head.i,
)
print('EOS')
The user dictionary files should be set to userDict field of sudachi.json in the installed package directory ofja_ginza_dict package.
Please read the official documents to compile user dictionaries with sudachipy command.
SudachiPy - User defined Dictionary
Sudachi User Dictionary Construction (Japanese Only)
GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models are distributed under the MIT License. You must agree and follow the MIT License to use GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models.
spaCy is the key framework of GiNZA.
SudachiPy provides high accuracies for tokenization and pos tagging.
Sudachi LICENSE PAGE, SudachiPy LICENSE PAGE, SudachiDict LEGAL PAGE, chiVe LICENSE PAGE
The GiNZA v5 transformer models (ja_ginza_electra and ja_ginza_bert_large) use Hugging Face Transformers as inference framework.
The parsing model of GiNZA v5 is trained on a part of UD Japanese BCCWJ r2.8 (Omura and Asahara:2018).
@inproceedings{omura-asahara-2018-ud,
title = "{UD}-{J}apanese {BCCWJ}: {U}niversal {D}ependencies Annotation for the {B}alanced {C}orpus of {C}ontemporary {W}ritten {J}apanese",
author = "Omura, Mai and
Asahara, Masayuki",
booktitle = "Proceedings of the Second Workshop on Universal Dependencies ({UDW} 2018)",
month = nov,
year = "2018",
address = "Brussels, Belgium",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/W18-6014/",
doi = "10.18653/v1/W18-6014",
pages = "117--125"
}
The named entity recognition model of GiNZA v5 is trained on a part of GSK2014-A (2019) BCCWJ edition (Hashimoto, Inui, and Murakami:2008). We use two of the named entity label systems, both Sekine's Extended Named Entity Hierarchy and extended OntoNotes5. This model is developed by National Institute for Japanese Language and Linguistics, and Megagon Labs.
ja_ginza_electra is a fine-tuned model of transformers-ud-japanese-electra-base-discriminator which is pretrained on more than 200 million Japanese sentences extracted from mC4.
The mC4 is published under the ODC Attribution License.
@article{2019t5,
author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
journal = {arXiv e-prints},
year = {2019},
archivePrefix = {arXiv},
eprint = {1910.10683},
}
ja_ginza_bert_large, ja_ginza_electra, and ja_ginza.ginza command enables GPU acceleration by default under certain conditions.
ginza -g -1.thinc-apple-ops on Mac OS running Apple Silicon are 3.10 through 3.12.ja_ginza_bert_largeginza-transformers was upgraded to v1.4.0.
spacy-transformers has a requirement of torch>=1.8.0 and transformers<4.53.3.ja_ginza_bert_largedoc.user_data["reading_forms"][token.i] -> token.morph.get("Reading")doc.user_data["inflections"][token.i] -> token.morph.get("Inflection")force_using_normalized_form_as_lemma(True) -> token.norm_ginza -m en_core_web_mdginza --require_gpu and ginza -g to take a gpu_id argument
gpu_id value is -1 which uses only CPUsginza -f json option always analyze the line which starts with # regardless the option value of -c. #215ginza -p {n_process} and ginzame) of ginza command #204Bunsetu span should not cross the sentence boundary #195Command Line -s option and set_split_mode() not working in v5.0.x #185ginzame not woriking in ginza ver. 5 #179Command Line -d option not working in v5.0.0 #178ja-ginza and ja-ginza-electra for -m option of ginza commandja-ginza-electra modelja-ginza model by adding morphologizer to the tail of spaCy pipelineginza package
pip install -U ginza ja-ginza-electrapip install -U ginza ja-ginzaCompoundSplitter and BunsetuRecognizer to compound_splitter and bunsetu_recognizer respectively-n
normalized_form as Token.lemma_-m (ja_ginza|ja_ginza_electra)
Degital_Game to Digital_Game-d option, which disables spaCy's sentence separator, to ginza command line toolginza command line tool works correctly without BunsetuRecognizer in the pipelineCompoundSplitter for nlp.to_disk()spacy.lang.ja of spaCy v2.3
Token.lemma_ with the output of SudachiPy's Morpheme.dictionary_form()ja_ginza_dict package safetyB-GPEToken._.* and add some entries for Doc.user_data[] and accessors
ginza.inflection(Token))ginza.reading_form(Token))ginza.bunsetu_bi_label(Token))ginza.bunsetu_position_type(Token))ginza.is_bunsetu_head(Token))from ginza import *)
Token._.sudachi field would be set after calling SudachipyTokenizer.set_enable_ex_sudachi(True), to avoid serializtion errorsimport spacy
import pickle
nlp = spacy.load('ja_ginza')
doc1 = nlp('This example will be serialized correctly.')
doc1.to_bytes()
with open('sample1.pickle', 'wb') as f:
pickle.dump(doc1, f)
nlp.tokenizer.set_enable_ex_sudachi(True)
doc2 = nlp('This example will cause a serialization error.')
doc2.to_bytes()
with open('sample2.pickle', 'wb') as f:
pickle.dump(doc2, f)
ja_ginza_dict from PyPIginza and ginzame
-i option to initialize the files of ja_ginza_dictginza and ja_ginza from PyPI
pip install ginza, and run ginzaja_ginza, is also available from PyPI.token.ent_type_ value is changed to Sekine's Extended Named Entity Hierarchy
ENE7 attribute to the last field of the output of ginzatoken._.ne
PHONE, EMAIL, URL, and PET_NAMEspacy pretrain over 100 epochs
spacy train effectively working on UD Japanese BCCWJSudachiDict_core-20191224ginzame
sudachipy by multiprocessing.Pool and output results with mecab like formatsudachipy command requires additional SudachiDict package installationginza (ginza.command_line.main_ginza)
mode to sudachipy_modedisable_pipes and recreate_correctorhash_comment, parallel, filesmecab to the choices for the argument of -f optionparallel NUM_PROCESS option (EXPERIMENTAL)ENE7 attribute to conllu miscellaneous field
ginza.ent_type_mapping.ENE_NE_MAPPING is used to convert ENE7 label to NEginzame (ginza.command_line.main_ginzame)
mecab like output formattoken._.ne for ner labelginza/sudachipy_tokenizer.py
SudachiTokenizer to SudachipyTokenizerSUDACHI_DEFAULT_SPLIT_MODE instead of SUDACHI_DEFAULT_SPLITMODE or SUDACHI_DEFAULT_MODEspacy to v2.2.3sudachipy to v0.4.2as_* type dependencies completelysplit_mode has been set incorrectly to sudachipy.tokenizer from v2.0.0 (#43)
split_mode incompatibility between the training phase and the ginza command.split_mode was set to 'B' for training phase and python APIs, but 'C' for ginza command.split_mode to 'C' entirely.-f and --output-format option to ginza command:
-f 0 or -f conllu : CoNLL-U Syntactic Annotation format-f 1 or -f cabocha: cabocha -f1 compatible formatbunsetu_index : bunsetu index starting from 0reading: reading of token (not a pronunciation)sudachi: SudachiPy's morpheme instance (or its list when then tokens are gathered by JapaneseCorrector)spacy pretrain command to capture the language model from UD-Japanese BCCWJ, UD_Japanese-PUD and KWDLC.-pt 'tag,dep' option of spacy trainginza command
ginza from the consoleginzaja_ginzaspacy.lang.ja is overridden by ginzasudachipy related directories
pip during ginza installationtoken._.bunsetu_bi_label, token._.bunsetu_position_typetoken._.infpos_detail (same value is set to token.tag_)$ git clone 'https://github.com/megagonlabs/ginza.git'
For normal environment:
$ python setup.py develop
Copy system.dic from installed package directory of ja_ginza_dict to ./ja_ginza_dict/sudachidict/.
The analysis model of GiNZA is trained by spacy train command.
$ python -m spacy train ja ja_ginza-4.0.0 corpus/ja_ginza-ud-train.json corpus/ja_ginza-ud-dev.json -b ja_vectors_chive_mc90_35k/ -ovl 0.3 -n 100 -m meta.json.ginza -V 4.0.0
GiNZA uses the pytest framework for testing, and you can run the tests via setup.py without install test requirements explicitly.
Some tests depends on the ginza parsing models, so install them before the tests is needed.
$ pip install ja-ginza ja-ginza-electra
$ pip install -e .
# full test
$ python setup.py test
# test single file
$ python setup.py test --addopts ginza/tests/test_analyzer.py
Overall
Maintained
Code review
PyPI
Latest release
Releases this year
2,848 followers · starred Jul 2021
156 followers · starred May 2020
228 followers · starred May 2024
859 followers · starred Jul 2020