Japanese NER: fine-tuned encoders vs. LLM in-context learning

How many labelled examples does it take before a fine-tuned 0.1B encoder overtakes an LLM doing in-context learning — and where does the remaining gap actually come from?

This repository answers that on Japanese named entity recognition (8 entity types, character spans), comparing four Japanese encoders fine-tuned for token classification against gpt-5-mini prompted zero-shot and few-shot.

Every number below is read from results/ and can be regenerated with python step7_final_compare.py. Nothing is transcribed by hand.

---

Headline results

---

Data

Eight labels: 人名 (person) 22.6% / 法人名 (company) 18.8% / 地名 (location) 16.4% / 製品名 (product) 9.2% / 政治的組織名 (political org) 8.9% / 施設名 (facility) 8.4% / その他の組織名 (other org) 8.0% / イベント名 (event) 7.7%.

Sentences are short (median 51 characters, max 296) and about 9% contain no entity at all — so "extract nothing" is a valid answer, and the LLM is explicitly instructed to return [].

The split reproduces llm-book's logic (random.seed(42), shuffle, 8:1:1) so the numbers line up with published baselines. Datasets are CC BY-SA 3.0 and CC BY 2.5 respectively; see their dataset cards for the authoritative terms.

---

Usage

Python 3.11 on arm64 (MPS requires it). Measurements below were taken on an Apple M4 Pro (48GB unified memory) with torch 2.13.0 / transformers 5.14.1.

python -m venv .venv-ner
source .venv-ner/bin/activate
pip install -r requirements.txt        # or requirements.lock.txt for exact pins

Step 0 — build data/. Run step0_prepare_data.ipynb first; it downloads both datasets and writes the four splits into data/. Everything else reads from there.

# Step 3 — fine-tune encoders (writes to models/ and results/)
python step3_train_encoder.py bert-ja-v3
python step3_train_encoder.py all                        # all four models
python step3_train_encoder.py --eval-only bert-ja-v3     # re-evaluate a saved checkpoint
python step3_train_encoder.py --train-size 200 bert-ja-v3    # a learning-curve point

# Step 4 — LLM in-context learning (needs OPENAI_API_KEY in the environment)
python step4_run_llm.py zero
python step4_run_llm.py few --shots 4
python step4_run_llm.py zero --eval-split wikinews_test --full

# Steps 5 & 7 — read results/ and print the report; no models or API calls involved
python show_results.py wikipedia_test_n200 --cost
python step5_show_curve.py
python step7_final_compare.py

The API key is read from the environment by the OpenAI SDK itself. If it only exists in your interactive shell profile, invoke through one:

zsh -i -c 'source .venv-ner/bin/activate && python step4_run_llm.py zero'

requirements.txt also lists mlx, mlx-lm and sentence-transformers; those were for a local-LLM branch that was dropped (see Not done) and are not needed to reproduce anything here.

---

How this is evaluated

Both sides are compared on character spans, never on BIO tags. The encoder output is converted BIO → character spans, and the LLM returns strings that are resolved to spans by searching the source text. This is what makes subword models, character-level models and a generative model comparable at all. seqeval is deliberately not used — it compares tag sequences, which the LLM does not have.

The metrics form a staircase, ordered by what we actually want to know. Each rung adds one condition, so where a model falls off tells you what kind of failure it is.

Plus two decomposition ratios: type accuracy = L3 TP / L2 TP ("of what was found, was the type right?") and span_exact = L5 TP / L3 TP ("of what was typed right, was the position right?").

Why L3 and not strict matching. Round-tripping gold spans through each tokenizer measures the ceiling that tokenization alone imposes. At L3 all schemes reach ≥0.9996, but at L5 they split hard: char-wwm 1.0000, bert-ja-v3 0.9959, deberta-v3 0.9887, and plain LUKE 0.7835 — because LUKE's vocabulary contains tokens like フランスの that swallow the following particle. Under strict matching, tokenizer granularity outweighs model ability and the comparison stops measuring what it claims to. L4 and L5 are still always reported.

⚠️ L3 and L4 are not nested. When the same string occurs several times in a sentence, L4 scores a hit even if the prediction points at the wrong occurrence, while L3 does not. For the encoders L4 = L5 (BIO pins the position), but for the LLM they diverge.

---

Results

1. Main comparison — same 200 sentences for every model

200 sentences / 524 entities drawn from wikipedia_test with seed 42. The LLM takes ~2 s per sentence, so the full 535 × 8 conditions was not affordable; the encoders were re-evaluated on the identical subset rather than compared across different footings.

The four encoders cannot be told apart — the intervals overlap heavily, and MPS training alone moves L3 by ~0.007 between identical re-runs. Only the encoder-vs-LLM gap is real.

The gap decomposes as: finding −0.084, typing −0.088, boundaries −0.023. The LLM's weakness is detection and label discipline; its boundaries are nearly as good as an encoder's. In fact luke-lite has by far the worst boundaries of anything here (span_exact 0.7397) — tokenizer granularity damages spans more than free-form generation does.

Per-label, the LLM is strong where general knowledge suffices (人名 F1 0.939) and weak exactly where the dataset's own conventions decide the answer: その他の組織名 0.534, 政治的組織名 0.622, 施設名 0.689. It over-produces events (イベント名 P 0.566 / R 0.909). It never hallucinated: 0 returned strings were absent from the input, and it never emitted an entity for a sentence that had none (empty-sentence accuracy 1.000). What it loses is recall — it found nothing at all in 8.8% of sentences that did contain entities.

† deberta-v3 alone runs with adam_epsilon=1e-4 instead of 1e-8. On MPS, AdamW collapses to NaN because denormals flush to zero, the second moment underflows, and the update blows up to m/eps; the collapse step moves monotonically with eps (1e-8 → step 2, 1e-6 → step 129, 1e-4 → stable). This is a handicap — read its ranking with that in mind.

2. Full test set (535 sentences), encoders only

3. Out of domain — Wikinews (178 sentences, full set)

The expectation going in was the opposite: an encoder fine-tuned on Wikipedia should be the brittle one, and a general LLM should transfer. It did not work out that way. The LLM's organisation labels collapse — その他の組織名 falls to F1 0.194 with 60 predictions against 12 gold spans — while 人名 holds at 0.913. General world knowledge transfers; a dataset's specific label boundaries do not.

Sentence-level recall drops from 0.912 to 0.812: the LLM finds nothing at all in 19% of sentences that contain entities.

4. Learning curve — bert-ja-v3, evaluated on the same 200 sentences

The ICL baseline is gpt-5-mini zero-shot at L3 0.7512 [0.708, 0.792].

Three things worth noting. It plateaus at 400 — a further 10x of data buys +0.044. At 100 examples, which sentences you happen to draw matters as much as anything else: the spread across three seeds is 0.046, comparable to the width of the confidence interval, so a single run at the left end means nothing. And below 50 examples ICL wins outright — that is the regime where prompting is the right tool.

Epochs scale with dataset size (~300 gradient updates, floored at 5 and capped at 30). With a flat 5 epochs, 25 sentences would get 5 updates total and you could not distinguish "too little data" from "never trained".

5. Cost

LLM: 1,972 ms and $0.000247 per sentence, no training. Encoder: 4.4 ms and $0 per sentence, after annotating 400 sentences and 275 seconds of training.

$247 to process a million sentences is not what stops you. Three weeks of wall clock is.

6. Few-shot did not help

On the held-out validation subset, varying shot count and demo-selection strategy moved L3 by 0.004 total (0.7908–0.7945) while raising cost 26–74%. On test, four conditions spanned 0.7512–0.7856 — a range that fits inside a single confidence interval. Use zero-shot with gpt-5-mini.

The honest version of this finding: two different explanations were constructed for differences of ~0.03 (first "the demos don't cover all labels", then "demo entity density steers output volume"), and both evaporated once intervals were computed. With 200 sentences the 95% interval for L3 is 0.05–0.07 wide. ner_utils.bootstrap_ci() exists because of that mistake, and evaluate() now stores per-sentence (TP, n_pred, n_gold) in results/detail.json so an interval can be computed later without re-running anything.

---

Hypotheses that turned out wrong

---

Caveats

• Measurement noise is larger than most of the differences discussed. 95% CI for L3 on 200 sentences: 0.05–0.07 wide. Repeated identical LLM runs: ~0.02. Encoder retraining with a fixed seed on MPS: ~0.007. Sentence sampling at 100 examples: 0.046.
• results/ keeps one row per run_id, so re-running a condition overwrites it. The gpt-5-mini zero-shot row is the most recent of several runs, not an average.
• MPS training is not reproducible even with seed=42; evaluation from a fixed checkpoint is exactly reproducible.
• deberta-v3 carries the adam_epsilon handicap described above. A fair comparison would need a CPU retrain at 1e-8.
• The split follows llm-book rather than grouping by Wikipedia page. 15 of 535 test sentences (2.8%) come from a page also seen in training; measured contamination is ~0.2% of entities.
• train_sec is absent for the 4,274-example encoder rows because those were re-evaluated with --eval-only, which overwrites the field. Curve rows retain theirs.

Not done

Local LLMs (Swallow-8B, llm-jp-4-8b) and LoRA fine-tuning of an LLM were dropped. Similarity- retrieval few-shot was skipped once fixed few-shot showed no benefit. Ranking the encoders against each other properly would need multiple seeds per model.

---

Layout

ner_utils.py               spans ⇄ BIO, span resolution, evaluate(), bootstrap_ci()
llm_icl.py                 prompt construction, response parsing, pricing
step0_prepare_data.ipynb   download, inspect, write data/
step1_span_bio_eval.ipynb  verify the conversion and the evaluation function
step2_pipeline.ipynb       one full lap with char-wwm
step3_train_encoder.py     fine-tune / evaluate the 4 encoders
step4_run_llm.py           run gpt-5-mini zero-/few-shot
step5_show_curve.py        learning curve with intervals
step7_final_compare.py     the report above
show_results.py            pretty-print results/summary.csv
results/                   summary.csv (one row per run) + detail.json (per-label, per-sentence)

The prompt in llm_icl.py is intentionally written in Japanese: it is part of the experimental setup, and translating it would change the results. Console output is likewise Japanese; code comments and documentation are English.