ModernBERT-Ja-310M Decision for LiteRT

argos1111/modernbert-ja-310m-jev is a Japanese cross-encoder fine-tuned from sbintuitions/modernbert-ja-310m. It reads one (question + state, candidate) pair and returns one logit. A softmax over the logits of one question's candidates gives the probabilities of a choice, a score level or a true/false answer. This repository holds it converted to LiteRT with litert-torch 0.9.3. The verified paths are the LiteRT CompiledModel Python API on desktop CPU and on the Apple Metal GPU with explicit FP32 computation (ai-edge-litert 2.1.6, Apple M4 Max), and the Kotlin CompiledModel API on the Galaxy S26 GPU with explicit FP32 computation (LiteRT 2.2.0). On the Galaxy S26 NPU, the shipped graphs pick the same candidate on every question. One probability moves by 0.0113, so the NPU misses the bar of |Δp| ≤ 0.01 on every answer by one question. On desktop CPU, the shipped graph (512-token window, float16 weights, float16 token table) and the reference (the source checkpoint in transformers 5.17.0 on CPU FP32) picked the same candidate on all 621 questions of 547 requests, and no probability differed by more than 0.00079. All measurements are from 2026-10-02.

On the Galaxy S26 GPU with explicit FP32, the same graph picked the same candidate on all 621 questions, with a maximum probability difference of 0.00079. The model makes one graph call per candidate. The warm median was 416.4 ms per pair [183.6, 669.0] and 2014 ms per request, over a 17-minute run that took the phone from thermal status 0 to 2.

The three requests are invented. The bars are real desktop-CPU output of modernbert_ja_310m_decision_s512_wfp16.tflite with the float16 token table.

Other formats: none found. A Hugging Face search on 2026-10-02 found no other port of the source model: no ONNX, GGUF, LiteRT or Core ML.

Files

In the wfp16 graphs only the weights are float16. Each of the 102 FULLY_CONNECTED weights feeds a DEQUANTIZE operator, so a wfp16 graph has 1,911 operators against 1,809 in the fp32 form. Activations stay float32.

Every graph holds the embeddings LayerNorm, the 25 encoder layers, the final LayerNorm and the classification head, and returns one logit for one pair. The host does the rest. It renders the texts, tokenizes each pair, looks up each token's table row, calls the graph once per candidate and applies the softmax.

Minimal usage

Python: desktop CPU

Needs numpy, tokenizers and ai-edge-litert (tested with 2.1.6). Torch and transformers are not needed. Run it from the directory that holds the downloaded files and decision_litert.py.

from decision_litert import DecisionModel

model = DecisionModel(
    "modernbert_ja_310m_decision_s512_wfp16.tflite",
    "tok_embeddings_fp16.bin",
    "tokenizer.json",
)
state = "先週届いた請求書に、解約済みのオプション料金が含まれています。差額の返金をお願いします。"
questions = {
    "department": {
        "type": "choice",
        "instructions": "担当部署は？",
        "criteria": {"billing": "請求・返金", "technical": "技術的な障害", "sales": "新規契約・見積"},
    },
    "refund": {"type": "noul", "instructions": "顧客は返金を要求しているか？"},
    "urgency": {
        "type": "score",
        "instructions": "対応の緊急度は？",
        "criteria": ["急がない", "早めに対応", "即時対応が必要"],
    },
}


def probs(answer):
    return ", ".join(f"{k} {p:.3f}" for k, p in answer["probabilities"].items())


answers = model.decide(state, questions)
department, refund, urgency = answers["department"], answers["refund"], answers["urgency"]
print(f"department: {department['choice']} (confidence {department['confidence']:.3f}; {probs(department)})")
print(f"refund: p(true) {refund['noul']:.3f}")
print(f"urgency: {urgency['score']:.3f} (confidence {urgency['confidence']:.3f}; {probs(urgency)})")
model.close()

On a Mac CPU it prints the following. The code rounds each value to 3 decimals.

department: billing (confidence 0.991; billing 0.999, technical 0.000, sales 0.001)
refund: p(true) 0.992
urgency: 1.221 (confidence 0.062; 0 0.170, 1 0.438, 2 0.392)

billing wins with probability 0.999 and confidence 0.991. The refund question gives p(true) = 0.992. The score answer is the expected zero-based level, so 1.221 falls between 早めに対応 (1) and 即時対応が必要 (2). Its confidence is 0.062 because the probability is spread over the three levels. On the same request (38 tokens in its longest pair), the reference gives the same answers, and no probability differs by more than 6.7e-05.

decide() returns one dict per question, keyed by the question's name. The choice and score answers also carry probabilities, and score carries a legend of the level texts. To run on the desktop GPU, pass accelerator="gpu". It sets explicit FP32 (GpuOptions(enforce_f32=True)), the Metal setting measured below.

Kotlin: Android GPU with explicit FP32

import android.util.Half
import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel
import java.io.File
import java.io.RandomAccessFile
import java.nio.ByteOrder
import java.nio.channels.FileChannel
import kotlin.math.exp

/** One graph call per (context, candidate) pair; ids come from a tokenizer that matches decision_litert.py. */
class DecisionGpu(dir: File, private val seq: Int = 512) : AutoCloseable {
  private val options = CompiledModel.Options(Accelerator.GPU).apply {
    gpuOptions = CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP32)
  }
  private val model =
    CompiledModel.create(File(dir, "modernbert_ja_310m_decision_s${seq}_wfp16.tflite").absolutePath, options, null)
  private val inputs = listOf("inputs_embeds", "attention_mask").associateWith { model.createInputBuffer(it, "serving_default") }
  private val outputs = mapOf("logit" to model.createOutputBuffer("logit", "serving_default"))
  private val table = RandomAccessFile(File(dir, "tok_embeddings_fp16.bin"), "r").use {
    it.channel.map(FileChannel.MapMode.READ_ONLY, 0, it.length()).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer()
  }
  private val embeds = FloatArray(seq * 768)
  private val mask = FloatArray(seq)

  /** ids of one pair: <s> context </s><s> candidate </s>, at most `seq` tokens (pad id 3 fills the rest). */
  fun logit(ids: IntArray): Float {
    require(ids.size <= seq)
    for (p in 0 until seq) {
      val row = (if (p < ids.size) ids[p] else 3) * 768
      for (c in 0 until 768) embeds[p * 768 + c] = Half.toFloat(table.get(row + c))
      mask[p] = if (p < ids.size) 1f else 0f
    }
    inputs.getValue("inputs_embeds").writeFloat(embeds)
    inputs.getValue("attention_mask").writeFloat(mask)
    model.run(inputs, outputs, "serving_default")
    return outputs.getValue("logit").readFloat()[0]
  }

  /** One question: the candidates' pair ids -> probabilities (softmax over the candidates). */
  fun probabilities(pairs: List<IntArray>): List<Double> {
    val z = pairs.map { logit(it).toDouble() }
    val top = z.max()
    val e = z.map { exp(it - top) }
    val sum = e.sum()
    return e.map { it / sum }
  }

  override fun close() {
    (inputs.values + outputs.values).forEach { it.close() }
    model.close()
  }
}

Create DecisionGpu once and call it from one worker thread. probabilities takes the pair ids of one question's candidates in candidate order and returns one probability per candidate. The ids must come from a tokenizer that matches decision_litert.py, and this repository has no Kotlin tokenizer. The block compiles with LiteRT 2.2.0 (AGP 8.9.1, Kotlin 2.2.21), but the block itself has not run on a device. The debug gate app that ran the Galaxy S26 rows below makes the same LiteRT calls. Its source is in android/gate/.

Host contract

decision_litert.py implements this contract. It follows the author's serving code: to_spec and modernbert/prompting.py for the rendering, answer_for for the read-out.

1. Render each question. The context is 質問: {instructions}\n状況: {state}. A state that is not a string becomes json.dumps(state, ensure_ascii=False). Each candidate is {label} or {label} — {description}, with U+2014 and one space on each side. For choice, the labels are the criteria keys and the descriptions their values (null allowed). For score, the labels are 0 to K-1 and the descriptions are the ordered level texts (2 to 10 levels). For noul, the labels are true and false (JSON spelling), and the descriptions are the optional true and false criteria texts.
2. Build one pair per candidate with tokenizer.json: <s> context </s><s> candidate </s>, ids 1 … 2 1 … 2. The pair budget is 512 tokens with truncation="only_first". The end of the context is cut, never the candidate.
3. Pad the ids on the right with id 3 to the window S (256 or 512), and fill the two float32 inputs. inputs_embeds [1,S,768] holds the table row of every position, the pad row included. attention_mask [1,S] is 1 for real tokens and 0 for padding. A pair longer than S raises an error.
4. Run the serving_default signature once per pair and read logit [1,1,1,1].
5. Apply softmax over one question's candidate logits, at temperature 1 with no calibration.
6. Read the answer. choice is the label with the highest probability, plus confidence. score is Σ i·pᵢ over the zero-based levels, plus legend and confidence. noul is p(true). confidence is 1 − H(p)/ln K, where K is the number of candidates. The author describes it as the concentration of the distribution, not calibrated correctness.

The tokenizers library on tokenizer.json, with enable_truncation(512, strategy="only_first"), gives the reference's ids on all 2,357 fixture pairs. The base model's tokenizer.json, which the author's serving code loads, gives the same ids on all 2,357. transformers 4.57.6 cannot load the repository's tokenizer class (TokenizersBackend), so the host uses the tokenizers library.

End to end, decision_litert.py with the S512 wfp16 graph and the float16 table gives the same answers as the reference on 107 of 107 requests (181 questions). They are the author's 16 items, the author's 12-question example, the 30 invented requests and 60 JGLUE requests. Probabilities differ by at most 0.00044, score values by 0.00033 and p(true) by 0.00047. With the S256 graph, it gives the same answers on 106 of 106 requests (178 questions).

Measured quality and performance

The reference is the source checkpoint run as the author's card snippet runs it: transformers 5.17.0 (the author's version), torch 2.12.1, sdpa attention, CPU FP32, 8 threads. Probabilities are the softmax over one question's candidates. "Same argmax" means the same winning candidate for a question. The author's card snippet prints 0.9996 for billing — 請求・返金. That is the author's number; our CPU FP32 run of the same snippet gives 0.9981.

The fixtures are 547 requests with 621 questions and 2,357 pairs. They hold 250 of the 2,508 JNLI rows and 250 of the 1,118 JCommonsenseQA rows of the JGLUE v1.3 test files (random.Random(0).sample, rendered as the author's tools/benchmark_jglue.py renders them). They also hold the author's 16 hand-written customer-support items (2 of them in English), the author's 12-question example and 30 Japanese requests invented for this run (10 customer support, 10 internal documents, 10 product FAQ). By kind there are 546 choice, 22 score and 53 noul questions. A boundary question is one where the reference's top probability is below 0.9. There are 123 of them. 546 of the 547 requests fit the 256-token window, and the S256 rows cover those 546 requests, with 618 questions (120 boundary).

The desktop rows ran on 2026-10-02 on one Apple M4 Max (16 cores, 128 GB), macOS 27.0, with ai-edge-litert 2.1.6 through the Python CompiledModel API. The CPU rows used 8 threads, with two gate jobs and a host-parity job running in parallel. The phone rows ran on 2026-10-02 on one Galaxy S26 (SM-S942Q, Android 16) with LiteRT 2.2.0 through the Kotlin CompiledModel API, in a debug gate app. Every row makes one graph call per pair. Times are informational. Each phone row states the thermal status and the battery temperature before → after, and the compile time. Phone times are warm medians per pair with [min, max].

S256 and S512 name the window. The _r4 rows use a variant of the same graph that keeps the attention matmuls as rank-4 BATCH_MATMUL. The shipped files lower them to rank 3. On the Mac, the _r4 rows give the same agreement numbers as the shipped files.

On the shipped S512 CPU row, every question kind matches: choice 546 of 546, score 22 of 22 and noul 53 of 53. The largest change in a score answer, the expected level, is 0.00033, and in p(true) it is 0.00047. The shipped S512 graph compiled in 0.13 s on CPU and in 2.34 s on Metal with explicit FP32.

With explicit FP32 on the S26 GPU, every operator ran in one LITERT_CL partition: 1,911 of 1,911 for the shipped files and 1,761 of 1,761 for the _r4 graph. On the NPU, both windows compiled whole as one DispatchDelegate node.

The S512 GPU row started on a cool phone (thermal status 0, 34.9 °C) and ran the 547 requests back to back for 17 minutes. Its minimum, 183.6 ms per pair, is the cool-phone number. Its median, 416.4 ms, covers the whole warming run. Every later row started hot, at thermal status 2 to 3 and 44.8 to 47.0 °C. The _r4 row ran hotter than the shipped S512 row, so the two medians are not a speed comparison. The three 60-request rows hold 60 fixture requests with one choice question each, 3 of them boundary, and ran after the other phone rows on a hot phone.

The NPU rows ran on the same hot phone. Their warm medians were 41.7 ms per pair at S256 and 142.6 ms at S512, against 229.2 and 416.4 ms on the GPU with explicit FP32. Per request, the medians were 1095 ms (GPU, S256), 2014 ms (GPU, S512), 192 ms (NPU, S256) and 706 ms (NPU, S512). The per-pair times cover writing the inputs, running and reading back. The gate app's Kotlin float16 table lookup is separate: a median of 16.9 ms per pair on the S256 GPU row and 26.4 ms on the S512 GPU row.

Accuracy on our fixtures comes from the reference on CPU FP32. On our JGLUE subset it is 238 of 250 JNLI questions (95.2%) and 233 of 250 JCommonsenseQA questions (93.2%). On the author's 16 hand-written items it is 14 of 16 (87.5%). One miss is a 3-level score question whose top two levels are 0.450 and 0.448. The other picks billing, at 0.72, for a password-reset mail. On desktop CPU, the shipped S512 graph picks the same candidate as the reference on every one of these questions. The invented requests are not in these counts.

The author reports 92.62% on JNLI (2323/2508), 92.40% on JCommonsenseQA (1033/1118) and 93.8% on the 16 items. Those are the author's numbers. The JGLUE numbers are on the full test files through the author's HTTP API, one question per request. The 16-item number is on the same 16 items. Ours are on a 250 + 250 subset of the same test files and on the same 16 items.

Not measured:

• Pixel phones: GPU, NPU and CPU.
• The Galaxy S26 CPU.
• The S26 GPU at default precision, FP16 and FP16_WITH_FP32_ACCUM beyond the 60-request S512 subset, and at S256 in those modes.
• The S26 NPU at S512 with the float32 table.
• Sustained throughput on a cool phone.
• The _r4 graph on the NPU, and at S256 on the phone.
• The Kotlin block itself on a device. The debug gate app makes the same calls.

fp16 and the GPU default precision

Use the GPU with explicit FP32 computation. It is the GPU setting verified here: GpuOptions(enforce_f32=True) on Metal and GpuOptions(precision = FP32) on the S26.

The shipped graphs carry two rewrites for fp16, and both keep fp32 results bit-identical. One is the mask constant. The key mask is (1 − attention_mask)·(−1e4), and the sliding-window band uses 0 / −1e4, instead of finfo.min. The other is SafeLayerNorm. It computes a LayerNorm on x·2⁻ᵏ with eps·2⁻²ᵏ. The shift k is 5 for layers 14–18 (attention and MLP norms), 6 for layers 19–24, 2 for the final norm and 0 elsewhere. Each k is ceil(log2(max|input|/64)), taken from the fixture pairs.

The reason is the size of the LayerNorm inputs. In fp32, on all 2,357 fixture pairs, they stay at or below 64 through layer 13. They reach 1,200–1,208 at layers 14–15, 1,997–2,011 at layers 16–18 and 2,629–2,694 at layers 19–24. LayerNorm squares its inputs, and the square of 2,694 is beyond the float16 range. The GeGLU products peak at 1,236, below 4,096, so the MLP needed no rescale. In PyTorch fp32, the graph with SafeLayerNorm is bit-identical to the same graph without it: the maximum difference is 0.0 on 7 probe pairs at S512, including the 351-token pair, and on 5 probe pairs at S256.

With these rewrites, fp16 computation still moves the answers:

• Metal at default precision (fp16 kernels), shipped S512 graph: 619 of 621 questions keep their winner. One flip is a JNLI question whose reference top-two gap is 0.02701. The other is one of the author's items, with a gap of 0.00177. 32 questions move by more than 0.01, and the largest probability change is 0.042.
• Metal at default precision, shipped S256 graph: 616 of 618 questions keep their winner. The flips have reference gaps of 0.02701 and 0.0045. 26 questions move by more than 0.01, and the largest change is 0.04.
• Galaxy S26 GPU at default precision and at FP16, shipped S512 graph, 60 JNLI requests with one choice question each (3 boundary), on a hot phone at thermal status 3: all 60 questions keep their winner, 2 move by more than 0.01, and the largest change is 0.0202. The two settings give the same numbers. Warm medians per pair on that hot phone were 231.0 ms at default precision and 252.7 ms at FP16. The explicit-FP32 S512 row started cool, so its median is not a speed comparison with these.
• Galaxy S26 GPU at FP16_WITH_FP32_ACCUM, same 60 requests: all 60 keep their winner, none moves by more than 0.01, and the largest change is 0.00446. Its warm median was 402.7 ms per pair on the same hot phone.

On the S26 NPU (Hexagon HTP, JIT, BURST mode), both shipped graphs keep the winner on all 618 and all 621 questions, boundary questions included. One noul question on an invented incident report moves from p(true) 0.2338 to 0.2451, a change of 0.0113. Its reference gap is 0.53, so it is not a near-tie. Every other question moves by 0.0062 or less. The NPU misses the bar (every answer within 0.01) by one question and 0.0013. With the float32 token table instead of the float16 one, the S256 graph on the NPU gives the same largest change, 0.01128, on the same question, so the table rounding is not what moves it. That run loaded the JIT cache in 0.5 s and had a warm median of 44.6 ms per pair [31.6, 54.1] at thermal status 3, 45.0 → 46.9 °C.

Limits

• The agreement numbers measure how closely the conversion follows the reference, not task accuracy. The conversion reproduces the reference's answers, including the wrong ones.
• The training data is Japanese. Per the author, languages other than Japanese and English are not evaluated. Our fixtures hold only 2 English requests, both among the author's 16 items.
• The pair budget is 512 tokens. The tokenizer cuts the end of the context, never the candidate. The S256 graph takes pairs up to 256 tokens, and decision_litert.py raises an error for a longer pair. In the fixtures, pairs have a median of 46 tokens, a p99 of 98 and a maximum of 351, and no pair reached the 512 budget.
• Each candidate is one graph call. A request costs one call per candidate across all its questions, so the time grows with the number of candidates. The 547 fixture requests took 2,357 calls.
• In the author's words, accuracy is not guaranteed outside the trained question types: NLI, intent, ordered scales, commonsense QA and true/false. Business domains such as customer support were not in the training data.
• The author reports that wrong answers can come with high probability: the mean top probability on JMMLU errors is 0.66. On two tasks outside training, the author reports 35.6% on livedoor 9-class news classification and 34.8% on JMMLU 4-choice, 500 items each.
• confidence is the concentration of the distribution, not calibrated accuracy, per the author. On the invented support message above, the urgency levels 早めに対応 and 即時対応が必要 get 0.438 and 0.392, and the confidence is 0.062.
• No image input, per the author.
• The checkpoint is F32, so the float16 token table is a rounding, not a copy. The largest rounding error is 6.1e-5, and 104 of the 78.6M values flush to zero. On desktop CPU, the S512 fp32 graph with the float16 table differs from the reference by up to 0.00039 in probability, against 9.8e-06 with the float32 table.
• The model does not generate text. It scores the candidates you give it.
• One desktop and one phone, the Galaxy S26, were tested. This repository has no Kotlin tokenizer.

Provenance, conversion and license

• Source: argos1111/modernbert-ja-310m-jev at revision 07cda23579443e7a33c0f474114279fa032340d6 (created 2026-09-19, one revision), published by the Hugging Face user argos1111. model.safetensors is 1,260,829,436 bytes, with 156 tensors, all F32, and 315,203,329 parameters, SHA-256 85ede5652889f5e4e114da5c6cdb923c31ae1f08cb0f5d0008a0631d6a4541d0. The token table, 102400 × 768, holds 78,643,200 of the parameters.
• Base model: sbintuitions/modernbert-ja-310m at revision 77675fc96a7e445e982e2ba90246b816efc74ec6, MIT.
• Model: ModernBertForSequenceClassification with 25 layers, hidden size 768 and 12 heads of size 64. The MLP is GeGLU (Wi 768→6144, Wo 3072→768). Every 3rd layer (layers 0, 3, …, 24: 9 layers) uses global attention with RoPE theta 160000. The other 16 layers use sliding-window attention over a window of 128 (±64 tokens) with RoPE theta 10000. LayerNorm has no bias (eps 1e-5), attention and MLP have no bias, GELU is exact (erf), the vocabulary has 102,400 tokens and max_position_embeddings is 8192. Layer 0 has no attention pre-norm.
• Head: the hidden state of position 0 (the <s> token) after the final LayerNorm, then a 768→768 dense layer without bias, GELU, a LayerNorm without bias and a 768→1 classifier with bias. That gives one logit per pair.
• Training data, per the author: JGLUE JNLI, JCommonsenseQA and JSTS train splits (CC BY-SA 4.0), JCoLA in-domain train (CC BY-SA 4.0), JCommonsenseMorality train (MIT) and MASSIVE 1.1 ja-JP train (CC BY 4.0). The test splits were not used. About 41% of the questions are true/false. The author trained on 94,384 questions (about 345,000 pairs) for 2 epochs with bf16 autocast. The author asks users to keep the credit to these datasets.
• The author states that the model is an independent, unofficial model, unrelated to any hosted decision API.
• Conversion: litert-torch 0.9.3 with torch 2.12.1 and transformers 4.57.6, at fixed shapes. conversion/graph.py (DecisionGraph) uses the transformers 4.57.6 ModernBERT modules with the control flow written out. RoPE comes from two baked cos/sin tables [1,1,S,64], one per layer type. Attention is q·kᵀ/8 plus an additive mask, then softmax and ·v. The final LayerNorm runs on position 0 only.
• fp16 rewrites: the −1e4 mask constant and SafeLayerNorm, both bit-identical in fp32 (see above). The graph differs from the stock ModernBertForSequenceClassification forward by at most 3.8e-6 in the logit, from the fp32 reduction order.
• Operators, in each fp32 graph: ADD 230, BATCH_MATMUL 50, CONCATENATION 50, FULLY_CONNECTED 102, GELU 26, MEAN 104, MUL 331, RESHAPE 383, RSQRT 52, SLICE 226, SOFTMAX 25, SQUARED_DIFFERENCE 52, SUB 53 and TRANSPOSE 125, plus 102 DEQUANTIZE in the wfp16 form. None is GATHER, GATHER_ND, CAST, SELECT_V2, BROADCAST_TO, MAXIMUM or EMBEDDING_LOOKUP. No tensor is int64 or above rank 4, and no BATCH_MATMUL has a constant left operand. The 4 RoPE constant buffers have 4 distinct contents.
• Float16 weights: ai-edge-quantizer 0.8.0, FLOAT_CASTING, weight-only, FULLY_CONNECTED, tensorwise.
• Token table: the checkpoint's F32 token embeddings rounded to the nearest float16 (ties to even), with no inf or NaN, SHA-256 60a9a140bf74a34a7717a4f8e9d1cb13d924742d5091146402049207dd6800b5. modernbert_ja_310m_decision_s512_wfp16.tflite has SHA-256 f87cda7c513d0b02ee2a98c76885cf048879e4c7e3d7e09e3ecde13e73cc61ec.
• Tokenizer: tokenizer.json has SHA-256 eab16bb632cc4eb35ad4f2664ae8e102cb0dfa1d49d6fc9e49541bc54616cb02. It is a Unigram model with Metaspace (prepend_scheme never), no normalizer and a TemplateProcessing pair <s> A </s><s> B </s>. <cls> (6) and <sep> (4) are in the vocabulary, but the pair template does not use them.
• Verification: the LiteRT CompiledModel Python API on desktop CPU and Metal, and the Kotlin CompiledModel API on the Galaxy S26. Every check is the same argmax plus the absolute probability difference against the reference.

License: CC BY-SA 4.0. The author released the weights under CC BY-SA 4.0. The converted graphs and the float16 token table are derivative works of those weights and carry the same license, share-alike. The conversion scripts, the Python host and the Kotlin block are released under the same terms. The base model is MIT. The license text is in LICENSE. Attribution, including the six training-data sources the author asks users to credit, is in NOTICE.