⚡ Nord v4.2 — Brain-Inspired Spiking Neural Network Language Model (140M)

The first SNN language model with spike-driven MoE, zonal specialization, and memory cortex — trained from scratch.

What's New in v4.2

Nord v4.2 is a complete architectural rebuild from v3. The key breakthrough: the model self-organizes into functionally distinct brain zones during training — sensory zones learn low firing rates, executive zones learn high firing rates, with no explicit supervision.

Model Description

Nord v4.2 is a 140M-parameter Spiking Neural Network (SNN) for text generation. It uses biologically-inspired Leaky Integrate-and-Fire neurons with membrane potentials, firing thresholds, and binary spikes. Unlike transformers where 100% of neurons activate per token, Nord activates only 3-9% — with different brain-inspired zones specializing in different functions.

Trained entirely from scratch — no transformer teacher, no distillation, no ANN-to-SNN conversion.

Key Features

Architecture

┌───────────────────────────────────────────────┐
│  Temporal Spike Encoder                       │
│  Token → 8 fast + 2 slow timestep currents    │
├───────────────────────────────────────────────┤
│  Sensory Zone (2 blocks)     rates: 8-10%     │
│  Standard FFN + LIF, feature extraction       │
├───────────────────────────────────────────────┤
│  Association Zone (2 blocks) rates: 10-14%    │
│  Spike-Driven MoE (4 experts, top-2) + LIF   │
├───────────────────────────────────────────────┤
│  Memory Cortex               rates: 0.5-1%    │
│  128 neurons, τ=0.99, gated temporal attn     │
├───────────────────────────────────────────────┤
│  Executive Zone (2 blocks)   rates: 11-26%    │
│  Standard FFN + LIF, decision & output        │
├───────────────────────────────────────────────┤
│  Readout (EMA over membrane potential)        │
│  → LM Head → vocabulary logits                │
└───────────────────────────────────────────────┘

Key Components

• Associative LIF Neurons — Learnable membrane time constants, voltage thresholds, synaptic currents, cascade amplification across 64 neural clusters
• ATan Surrogate Gradient — Differentiable spike function for backpropagation
• Spike-Driven MoE — Expert routing based on cluster spike-rate activity, not dense networks
• Memory Cortex — Persistent slow memory with multi-head temporal attention readout
• Adaptive Spike Regulator — Asymmetric homeostasis: penalizes too-low firing 3x more than too-high, anti-death floor at 1%
• RoPE — Rotary position embeddings for sequence position encoding
• Synaptic Resonance Attention — Temporal mixing over spike patterns (not naive flattening)

Model Configuration

d_model: 496
n_heads: 8
n_layers: 6 (2 sensory + 2 association + 2 executive)
d_ff: 1024
n_experts: 4
top_k_experts: 2
memory_size: 128
T_fast: 8, T_slow: 2
max_seq_len: 512
vocab_size: 128,256
tokenizer: Llama-3.2 (meta-llama/Llama-3.2-1B)

Emergent Zonal Specialization

The most significant finding: the model self-organizes functionally distinct zones during standard training. No manual assignment, no hardcoded rates.

Zone              Spike Rate    Biological Analog
─────────────────────────────────────────────────────
Sensory           8-10%         Primary sensory cortex
Association       10-14%        Parietal/temporal cortex
Memory Cortex     0.5-1%        Hippocampus (selective)
Executive [0]     11-15%        Premotor cortex
Executive [1]     22-26%        Prefrontal cortex
─────────────────────────────────────────────────────

This mirrors biological cortical organization where prefrontal cortex has higher baseline activity than sensory cortex.

Training

• Dataset: ~2.2M text samples, general English corpus
• Hardware: NVIDIA A5000 24GB (rented on Vast.ai)
• Optimizer: AdamW (lr=3e-4 → 1e-5 cosine decay, weight_decay=0.01)
• Batch size: 2 × grad_accum=16 (effective 32)
• Sequence length: 512

Loss Progression

Parameter Breakdown

Usage

import torch
from nord_core_v4 import NordConfig, NordModel
from transformers import AutoTokenizer

# Load
ckpt = torch.load("nord_v4_latest.pt", map_location="cuda")
cfg = NordConfig(**ckpt["config"])
model = NordModel(cfg).cuda()

# Filter persistent state buffers (size varies with batch)
state = {k: v for k, v in ckpt["model_state_dict"].items()
         if "_v_mem_state" not in k and "_i_syn_state" not in k}
model.load_state_dict(state, strict=False)
model.eval()

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")

Or use the interactive chat:

python chat_v4.py
# Commands: /stats, /memory, /expert, /stdp on|off, /reset, /quit

Generation Examples

Step 3,600 (loss 5.5) — no coherence:

"Queen was being too late. The lake is not to be found in a variety of birds and stynesan trees."

Step 29,000 (loss 4.5) — topic understanding, broken logic:

"The internet is equipped with computers that harness data from television and radio vehicles. Its central and large uses can help business use and share information on devices and systems."

Step 39,000 (loss 4.3) — thematic coherence, real entities:

"A cybersecurity campaign that uses a computer science machine learning robot to guide players, and has refined algorithms. The popular game research software made by OpenAI security researchers..."

Spike Dynamics

Sparsity is dynamic and input-dependent — the model recruits more neurons for harder inputs, just like a biological brain.

Comparison with Other SNN Language Models

Version History

Limitations

• Text quality not competitive with GPT-2 at same parameter count (loss 4.3 vs ~3.0)
• Coherence degrades after 2-3 sentences at 140M scale
• Multilingual leakage in long generations (dataset artifact)
• Scaling beyond 140M untested for v4.2
• No formal benchmark evaluation yet
• Hallucination present

Scaling Hypothesis

If zonal specialization persists at scale, an 86B SNN could potentially:

• Match 86B transformer quality
• Run inference with compute of a 3-4B dense model (96% sparsity)
• Deploy on neuromorphic hardware (Intel Loihi) with orders of magnitude energy savings

This is unproven. The roadmap: 140M → 500M → 1-2B, testing at each scale.

Citation

@software{nord2026,
  title={Nord v4.2: Brain-Inspired Spiking Neural Network Language Model with Spike-Driven MoE and Zonal Specialization},
  author={Zemondsa},
  year={2026},
  url={https://github.com/zemondsa/nord-ai}
}

About

Built solo by an 18-year-old Ukrainian student studying electronics in Norway. No PhD, no team, no funding — just a rented A5000 and curiosity.