Thistle is a React Native Nitro Module for running local GGUF language models on-device. It keeps the JavaScript API small while the native runtime handles tokenization, llama.cpp inference, CPU execution, and iOS Metal acceleration.

Features

• Local GGUF inference from a model bundled into the native iOS app
• iOS Metal acceleration with CPU fallback through llama.cpp
• Instruction scaffolding with the built-in eggwhite preset or custom guidance
• Structured context injection for app data, attachments, and retrieved content
• Independent model instances so apps can manage multiple local models
• Typed runtime limits for input, context, output, and reasoning tokens
• Debug logging controls for prompt/output diagnostics during development

Requirements

• React Native with Nitro Modules
• iOS 15.1 or newer for the current iOS implementation
• A GGUF model compatible with the bundled llama.cpp runtime
• Enough device memory for the model, context state, and Metal buffers

Thistle is a React Native package for simple, local AI inference. The intended developer experience is:

1. Install the package.
2. Drop a supported model into assets/.
3. Initialize one or more model instances.
4. Use those instances anywhere in your JavaScript or TypeScript code.

The public API is designed to hide Nitro Modules, native ML runtimes, tokenizers, delegates, and hardware selection from the application developer.

Installation

npm install react-native-thistle react-native-nitro-modules

react-native-nitro-modules is required because Thistle uses Nitro Modules for its native bridge.

For an existing React Native app, install the iOS dependencies after installation:

cd ios
pod install
cd ..

The package postinstall script creates an assets/ directory in the consuming app when it does not already exist. To opt out:

THISTLE_SKIP_ASSETS=1 npm install react-native-thistle

Creating the folder is only a convenience. You still need to add a model file yourself.

Add A Model

Put a GGUF model in the application-level assets/ directory:

my-app/
	assets/
		model.gguf
	src/
		chat.ts

For iOS, CocoaPods automatically copies .gguf files from the application-level assets/ directory into the app bundle during the build. For Android, copy the model into the application's android/app/src/main/assets/ directory, or add an equivalent Gradle copy task. Both platforms can initialize the model by filename:

const model = await Thistle.init('model.gguf');

The example Metro configuration blocks .gguf files from the JavaScript bundle, which avoids Metro's asset-size limits for large models. If your app has its own Metro setup and uses a small static asset with require('./assets/model.gguf'), Thistle also accepts the resulting React Native asset reference. Dynamic paths such as require(pathFromUserInput) cannot be resolved by Metro. Set THISTLE_MODEL_DIR during the iOS build if your models live in a different directory.

Initialize And Prompt

import { Thistle } from 'react-native-thistle';

const model = await Thistle.init('model.gguf', {
  scaffolding: 'eggwhite',
  maxInputTokens: 0,
  maxContextSize: 0,
  maxOutputTokens: 0,
  maxReasoningTokens: 0,
  debug: false,
});
const answer = await model.prompt('What is the capital of France?');

console.log(answer);

scaffolding is optional persistent guidance applied to every prompt. The built-in eggwhite preset provides conservative assistant behavior, evidence-aware answers, arithmetic checking, and attachment handling. You can also pass your own scaffolding string. maxInputTokens, maxContextSize, maxOutputTokens, and maxReasoningTokens are numeric runtime limits. Falsy values such as 0, undefined, and null mean no explicit limit; the model's own context capacity remains the hard upper bound. Each setting can be overridden for an individual prompt.

Runtime Options

The numeric limits can be passed to Thistle.init() as defaults or overridden in model.prompt(text, options). Falsy limits are normalized to no explicit limit; native model/context capacity remains the hard ceiling.

Prompt-specific data can be injected with the second argument to prompt():

const answer = await model.prompt('Summarize the latest result.', {
  data: { result: 'The build passed', timestamp: Date.now() },
  maxOutputTokens: 200,
});

Debug logging is enabled by default. Set debug: false in the initialization options to silence Thistle's prompt, output, and failure logs.

Thistle.init() accepts either a Metro asset returned by require(...) or a string path supported by the native runtime:

const model = await Thistle.init('model.gguf');

Each call creates an independent model instance. Release instances when they are no longer needed:

const model = await Thistle.init('model.gguf');

try {
  const answer = await model.prompt('Summarize this text.');
  console.log(answer);
} finally {
  model.unload();
}

Multiple Models

Multiple models can be initialized with separate handles. This is useful for different workloads, but every loaded model consumes memory:

import { Thistle } from 'react-native-thistle';

const chatModel = await Thistle.init('chat.gguf');
const smallModel = await Thistle.init('small-model.gguf');

const answer = await chatModel.prompt('Write a short welcome message.');
const quickAnswer = await smallModel.prompt('Reply with one word.');

chatModel.unload();
smallModel.unload();

Prefer a smaller quantized model when the device has limited memory. A model can require substantially more memory than the size of its file because of runtime buffers and context state.

Example Chat App

The repository example app demonstrates the intended application shape in example/src/App.tsx. Its model is defined at:

const modelAsset = 'Qwen3VL-2B-Instruct-Q4_K_M.gguf';

The example initializes that bundled model on mount, displays model status, accepts text input, invokes model.prompt(...), and renders assistant output with Markdown and native SVG LaTeX. Responses are selectable and can be copied to the clipboard.

Run the example from the repository root with two terminals:

# Terminal 1
yarn example start

# Terminal 2
yarn example ios

For Android project work:

yarn example android

The example uses the Yarn workspace. npm run example by itself is incomplete because it forwards to a workspace command. The equivalent npm commands are:

npm run example -- start
npm run example -- ios
npm run example -- android

Current Implementation Status

The JavaScript API, iOS llama.cpp bridge, Metal backend, bundled shader resources, and chat harness are in place:

type ThistleModel = {
  prompt(text: string): Promise<string>;
  unload(): void;
};

Android inference uses the same llama.cpp runtime through a JNI bridge. The current Android path supports bundled GGUF files and CPU/GPU backend selection through llama.cpp; test on a physical arm64 device for realistic model size and performance.

The intended native architecture is:

React Native JavaScript
				|
				v
Thistle.init() / model.prompt()
				|
				v
Nitro Module
				|
				v
llama.cpp GGUF runtime
				|
				+-- iOS Metal / CPU
				+-- Android llama.cpp backends

The example's Android Gradle build copies example/assets/*.gguf into the APK assets automatically. For a separate consumer app, add the GGUF to that app's Android assets and use a physical arm64 device when validating prompt() with a large model.

Development Workflow

From the repository root:

yarn
yarn nitrogen

Run Nitrogen whenever src/Thistle.nitro.ts changes. Native changes require rebuilding the example app. JavaScript changes are picked up by Metro.

Validate the package with:

yarn typecheck
yarn test
yarn lint

The root lint configuration excludes the vendored llama.cpp sources. Their web UI has a separate Prettier configuration and dependency set, so it is not included in Thistle package linting.

Contributing

• Development workflow
• Sending a pull request
• Code of conduct

License

MIT

---

Made with create-react-native-library