Leverage CTC-WS and Multi-Stage Prompting for Robust In-Vehicle Spoken Language Understanding

This repository contains the code implementation of our graduation project (still researching):

Robust Real-Time Automatic Speech Recognition and Natural Language Understanding in Low-Resource Environments. Po-Yen Chen, Kai-Chen Tsai.

This work evaluates Spoken Language Understanding (SLU) systems in real-world environments. We leverage a CTC-based Word Spotter (CTC-WS) to enhance ASR performance, and adopt a multi-stage prompting strategy inspired by GPT-SLU to improve the NLU module. This project is a work in progress, and we are actively exploring more robust methods for real-time SLU. Additionally, we investigate challenging scenarios such as multi-intent detection and multilingual understanding.

This code will generate a temporary demo website by Gradio. You can use the website to understand how we do.

Architecture

.

Results

Experiment Setting

• Model:
  • ASR: Use parakeet-ctc-0.6b.
  • NLU: Use Gemma3
• Dataset:
  • SLURP
• Baseline:
  • GPT-SLU

Preperation

CUDA

Our code can run on CUDA 12.8.

Python Package

We use uv as package system. You can use this code to setup the environment.

uv sync

ASR model

Run this code to download the stt_en_fastconformer_ctc_large, the model we used in our code.

uv run load.py

NLU model

You need pull these modle if you want to uses by Ollama.

ollama pull gemma3:4b
ollama pull gemma3:12b
ollama pull gpt-oss:20b

We will use ollama to access these model. Make sure you can use ollama properly.

Usage

Use this command to build the website by Gradio.

uv run main.py

Then you will get the links of website.

License

Copyright (c) 2025 Po-Yen Chen and Kai-Chen Tsai
All rights reserved.

This source code is not licensed for redistribution or use.