DeepUnity (2022.3.43f1 lts)

Reference Papers

DeepUnity is an add-on framework that provides tensor computation [with GPU acceleration support] and deep neural networks, along with reinforcement learning tools that enable training for intelligent agents within Unity environments using state-of-the-art algorithms.

Run your first DeepUnity script

using UnityEngine;
using DeepUnity;
using DeepUnity.Optimizers;
using DeepUnity.Activations;
using DeepUnity.Modules;
using DeepUnity.Models;

public class Tutorial : MonoBehaviour
{
    [SerializeField] private Sequential network;
    private Optimizer optim;
    private Tensor x;
    private Tensor y;

    public void Start()
    {
        network = new Sequential(
            new Dense(512, 256, device: Device.GPU),
            new Swish(),
            new Dropout(0.1f),
            new Dense(256, 64, device: Device.GPU),
            new Swish(),
            new RMSNorm(64),
            new Dense(64, 32)).CreateAsset("TutorialModel");
        
        optim = new AdamW(network.Parameters());
        x = Tensor.RandomNormal(64, 512);
        y = Tensor.RandomNormal(64, 32);
    }

    public void Update()
    {
        Tensor yHat = network.Forward(x);
        Loss loss = Loss.MSE(yHat, y);

        optim.ZeroGrad();
        network.Backward(loss.Grad);
        optim.Step();

        print($"Epoch: {Time.frameCount} - Train Loss: {loss.Item}");
        network.Save();
    }
}

Large Language Models

DeepUnity provides full-GPU inference (HLSL compute shaders, streaming weight upload, KV caching) for the following LLMs, along with SFT scripts w/ QLoRA:

All weights are exported into the engine's own format with import_params.py (--quant fp16|int8|int4) into Assets/Resources/Weights/. Every ported model registers itself in LLMRegistry, so it automatically shows up in the NPC inspector dropdowns.

Behaviour script overriding example

using UnityEngine;
using DeepUnity;
using UnityEngine.UI;

public class ChatWithQwen : MonoBehaviour
{
    [SerializeField] private Text display;
    [SerializeField] private Text uiText;
    [SerializeField] private float temperature = 0.7f;
    [SerializeField] private float top_p = 0.95f;
    private Qwen3_5ForCausalLM model;
    public void Start()
    {
        // INT8 weights: ~half the VRAM/disk of FP16, ~lossless quality. Export with
        // `python import_params.py Qwen/Qwen3.5-0.8B --quant int8`.
        model = new Qwen3_5ForCausalLM(Qwen3_5Size.B0_8, LLMQuant.INT8);

        StartCoroutine(model.Generate(
            prompt: "Explain quantum entanglement in just 5 words.",
            onTokenGenerated: (x) => {  display.text += x;  },
            temperature: temperature,
            top_p: top_p));
    }
}

You can finetune your own SLMs with the HuggingFace trainer (full or with adapters) for role-playing or other content generation, [merge the base model with the adapters,] then use the import_params.py script to replace the /params folder with the serialized new weights (--quant fp16|int8|int4).

Text To Speech

DeepUnity runs TTS fully on the GPU as well, with real-time streaming synthesis (speech starts while the reply is still generating) and per-NPC voices:

Speech-to-text runs fully on the GPU too: Qwen3-ASR 0.6B/1.7B (RTF 0.22-0.66) and Parakeet-TDT 0.6B v2/v3 (RTF ~0.08), both validated transcript-exact against their reference implementations.

Speaking through a PocketTTSVoice component

using UnityEngine;
using DeepUnity;

// Attach next to an AudioSource. The engine is shared across all voices in the
// scene; weights stream to the GPU budgeted, without frame drops.
public class TalkingNpc : MonoBehaviour
{
    [SerializeField] private PocketTTSVoice voice; // voiceName, pitch set in the inspector
    [SerializeField] private AudioClip referenceClip; // ~10 s clip to clone a voice at runtime

    public void Start()
    {
        // one-shot utterance
        voice.Say("Welcome, traveler! What brings you to the village?");

        // or clone a voice from a reference clip instead of a baked voiceName
        // voice.SetClonedVoice(referenceClip);
    }

    // ...or stream an LLM reply as it generates: feed token deltas and flush at the end.
    // Each completed sentence is synthesized and played while the next one is still being written.
    public void OnLlmToken(string delta) => voice.FeedText(delta);
    public void OnLlmDone() => voice.FlushText();
}

Voices can be auditioned in the VoiceLab scene (menu DeepUnity/TTS/Build VoiceLab Scene): pick an engine + voicepack, tune pitch/speed live and save presets.

Game NPCs

NPCChatBase is a drop-in MonoBehaviour that turns the LLM + TTS stack into a talking game character — everything is set up in the inspector (persona, model, voice), the demo subclasses (NPCInteractor3D/NPCInteractor2D) only add presentation.

Features:

• Per-NPC persona (system prompt), any registered LLM + quantization, sampling overrides, optional <think> reasoning with an animated Thinking… indicator.
• Spoken replies through any TTS engine, per-NPC voice or cloned from an AudioClip, audio-synced talk animation (or text-only mode).
• Prefetch zones: model weights stream to the GPU as the player approaches, released on exit; NPCs sharing a model share one GPU instance.
• Auto frame pacing (60+ fps while the model works) with a per-NPC Smooth ⇄ Speed dial.
• Conversation memory driven by Max Context Length: reset-on-close, persistent with disk KV snapshots, or self-compacting history (HISTORY: block in the system prompt) that lets a chat continue indefinitely.
• Escape-interrupt, chat window sharing across NPCs, StartInteraction()/AskNPC()/CloseInteraction() API.

Reinforcement Learning

In order to work with Reinforcement Learning tools, you must create a 2D or 3D agent using Unity provided GameObjects and Components. The setup flow works similary to ML Agents, so you must create a new behaviour script (e.g. ReachGoal) that must inherit the Agent class. Attach the new behaviour script to the agent GameObject (automatically DecisionRequester script is attached too) [Optionally, a TrainingStatistics script can be attached]. Choose the space size and number of continuous/discrete actions, then override the following methods in the behavior script:

• CollectObservations()
• OnActionReceived()
• Heuristic() [Optional]
• OnEpisodeBegin() [Optional]

Also in order to decide the reward function and episode's terminal state, use the following calls:

• AddReward(reward)
• SetReward(reward)
• EndEpsiode()

When the setup is ready, press the Bake button; a behaviour along with all neural networks and hyperparameters assets are created inside a folder with the behaviour's name, located in Assets/ folder. From this point everything is ready to go.

To get into advanced training, check out the following assets created:

• Behaviour can be set whether to use a fixed or trainable standard deviation for continuous actions. Inference and Training devices are also available to be set (set both on CPU if your machine lacks a graphics card). TargetFPS modifies the rate of physics update, being equal to 1 / Time.fixedDeltaTime (default: 50).
• Config provides all hyperparameters necesarry for a custom training session.

Behaviour script overriding example

using UnityEngine;
using DeepUnity.ReinforcementLearning;

public class MoveToGoal : Agent
{
    public Transform apple;

    public override void OnEpisodeBegin()
    {
        float xrand = Random.Range(-8, 8);
        float zrand = Random.Range(-8, 8);
        apple.localPosition = new Vector3(xrand, 2.25f, zrand);
        
        xrand = Random.Range(-8, 8);
        zrand = Random.Range(-8, 8);
        transform.localPosition = new Vector3(xrand, 2.25f, zrand);
    }

    public override void CollectObservations(StateVector sensorBuffer)
    {
        sensorBuffer.AddObservation(transform.localPosition.x);
        sensorBuffer.AddObservation(transform.localPosition.z);
        sensorBuffer.AddObservation(apple.transform.localPosition.x);
        sensorBuffer.AddObservation(apple.transform.localPosition.z);
    }

    public override void OnActionReceived(ActionBuffer actionBuffer)
    {
        float xmov = actionBuffer.ContinuousActions[0];
        float zmov = actionBuffer.ContinuousActions[1];

        transform.position += new Vector3(xmov, 0, zmov) * Time.fixedDeltaTime * 10f;
        AddReward(-0.0025f); // Step penalty
    } 

    private void OnTriggerEnter(Collider other)
    {
        if (other.CompareTag("Apple"))
        {
            SetReward(1f);
            EndEpisode();
        }
        if (other.CompareTag("Wall"))
        {
            SetReward(-1f);
            EndEpisode();
        }
    }
}

This example considers an agent (with 4 space size and 2 continuous actions) positioned in the middle of an arena that moves forward, backward, left or right, and must reach a randomly positioned goal (see GIF below). The agent is rewarded by 1 point if he touches the apple, and penalized by 1 point if he's falling of the floor, and in both situations the episode ends.

Tips

• Parallel training is one option to use your device at maximum efficiency. After inserting your agent inside an Environment GameObject, you can duplicate that environment several times along the scene before starting the training session; this method is necessary for multi-agent co-op or adversarial training. Note that DeepUnity dynamically adapts the timescale of the simulation to get the maximum efficiency out of your machine.

• In order to properly get use of AddReward() and EndEpisode() consult the diagram below. These methods work well being called inside OnTriggerXXX() or OnCollisionXXX(), as well as inside OnActionReceived() rightafter actions are performed.

• Decision Period high values increases overall performance of the training session, but lacks when it comes to agent inference accuracy. Typically, use a higher value for broader parallel environments, then decrease this value to 1 to fine-tune the agent.

• Input Normalization plays a huge role in policy convergence. To outcome this problem, observations can be auto-normalized by checking the corresponding box inside behaviour asset, but instead, is highly recommended to manually normalize all input values before adding them to the SensorBuffer. Scalar values can be normalized within [0, 1] or [-1, 1] ranges by using the formula normalized_value = (value - min) / (max - min). Note that inputs are clipped for network scability (default [-5, 5]).

• The following MonoBehaviour methods: Awake(), Start(), FixedUpdate(), Update() and LateUpdate() are virtual. If neccesary, in order to override them, call the their base each time, respecting the logic of the diagram below.

---

Training on built application for faster inference

• Training inside the Editor is a bit more cumbersome comparing to the built version. Building the application and open it up to start up the training enables faster inference, and the framework was adapted for this.

• Whenever you want to stop the training, close the .exe file. The trained behavior is automatically saved and serialized in .json format on your desktop. Go back in Unity and check your behavior asset, and press on the newly button to overwrite the editor behavior with the trained weights from .json.

• The previous built app, along with the trained weights in .json format are now disposable (remove them and replace the build with a new one).

---

Sorter agent whose task is to visit the tiles in ascending order

These crawlers are training to scrape over the internet

And not only the crawlers

All tutorial's scripts are included inside Assets/DeepUnity/Tutorials folder, containing all the features provided by the framework and RL environments inspired from ML-Agents examples (note that not all of them have trained models attached).