Datus · Open-Source Data Engineering Agent

Website · Docs · Quick Start · Dosi · Release Notes

English | 简体中文

---

Datus is the open-source data engineering agent for the modern data stack: one agent that connects your warehouse, catalog, semantic layer, and BI, grounded in an evolvable context engine your team owns.

Datus handles SQL authoring and validation, semantic model and metric construction, and the generation of pipelines, reports, and dashboards. Every run and every correction settles into context, which steadily raises the accuracy of its output. The whole stack stays open and flexible: databases, BI, schedulers, LLMs, and your team's own tools all connect through standard interfaces.

Architecture

The diagram reads top to bottom: who uses Datus, what the agent is made of, and what it connects to.

• Three entry points, by role: data engineers work in Datus-CLI to explore data and build assets; analysts ask through Datus-Chat on the web, in Slack/Feishu, or in VS Code, and their feedback flows back into the agent; other agents and applications consume Datus-API over REST and MCP.
• The agent core: subagents package curated context, tools, and rules for one business domain, and skills add packaged tools. Underneath sits the context engine: metadata, metrics, reference SQL, knowledge, and local files, retrieved through business-domain trees plus vector search, with storage on embedded LanceDB and SQLite and PostgreSQL for teams that share context.
• Connected systems: LLM providers, data warehouses, job schedulers, BI tools, and MCP servers and clients connect through adapters and plugins. The Dosi semantic engine is built into the agent.

Features

Semantic layer

• Automated semantic modeling: the agent reads your database schema and SQL history, then generates OSI semantic models and metric definitions, with no hand-written YAML.
• Dosi execution engine: compiles one semantic model into SQL for 13+ database dialects, and ships as an independent program you can also run as a CLI, REST server, or MCP server.
• Metric Q&A and attribution: AskMetrics answers business questions from metric definitions instead of improvising SQL, and when a metric moves, dimension attribution locates which dimension drove the change.

Agent and context

• Sharper with use: the context engine gathers schemas, reference SQL, and business rules, and writes every correction back, so later answers keep getting more accurate.
• Subagent delivery: curate context, tools, and rules for one domain, package them as a dedicated chatbot, and serve it to analysts over web, API, MCP, Slack/Feishu, or VS Code.
• Data engineering automation: built-in subagents handle cross-database migration, ETL job generation, and wide-table builds, with Airflow orchestration and Superset/Grafana dashboard read-write.
• Report and dashboard generation: produce self-contained HTML reports and interactive dashboards straight from chat, previewed locally with no SaaS backend.

Openness and governance

• Open ecosystem: adapters for 21 databases, 10+ LLM providers, and an MCP server and client.
• External integrations: the plugin framework connects third-party platforms and in-house tools to the agent; one datus-plugin.yml manifest declares CLI commands, skills, and prompt context, with per-project activation.
• Skills: packaged tools following the agentskills.io convention, installable from a marketplace.
• Enterprise governance: tiered permission profiles, statement-level SQL authorization with AI pre-review, bash confined to an OS-level sandbox, and traces exportable to any OTLP platform.

Quickstart

Linux x86_64/aarch64 (glibc 2.28+) or Apple Silicon macOS:

curl -fsSL https://raw.githubusercontent.com/datus-ai/datus-agent/main/install.sh | sh

Open a new shell and run datus, then:

1. /model to configure an LLM
2. /datasource to add a datasource
3. /init (optional) to scan the current project

Manual install works too: pip install datus-agent (Python 3.12+ on the supported platforms above); more install options are covered in the Quickstart. When pip spends minutes backtracking through litellm releases (versions up to 0.3.9 are affected), uv resolves the same set in seconds: pip install uv && uv pip install datus-agent --system. The end-to-end tutorial demonstrates the full flow on a sample dataset. Configuration has two levels: a global agent.yml for the main settings, and a per-project .datus/config.yml for overrides such as the active model and default datasource (see the configuration docs).

Interfaces

The examples below use a datasource named demo; create one first with /datasource.

Tip: Print mode streams JSON to stdout for scripting and CI: datus -p "your question" --datasource demo.

Development

Developing Datus

Start here to work on Datus itself: install dependencies with uv, then run the PR test harness and format checks before submitting.

uv sync                                                                    # Install dependencies
uv run python ci/run-pr-tests.py upstream/main                             # PR CI harness (no external deps)
uv run ruff format datus/ tests/ && uv run ruff check --fix datus/ tests/  # Lint & format

See CLAUDE.md for development conventions, architecture patterns, and testing rules.

Developing a plugin

Extending Datus does not require touching its core: declare CLI commands, skills, and prompt context in a datus-plugin.yml manifest, then pack it for distribution and per-project activation. The plugin development guide walks through the full flow.

License

Apache 2.0