---

Quick links

---

What is Duckle?

An open-source ETL platform you run on your own infrastructure. Drag sources, transforms, validators and sinks onto a canvas, wire them together, and press Run. Duckle compiles the graph to SQL and executes it on a real columnar engine, with live previews, the generated SQL visible on every node, and no hidden state.

You build a pipeline on a laptop and deploy that same file to a server, where it runs on a schedule under a web console with roles, alerts and an audit log. Nothing is rewritten in between, and nothing is metered.

In short: a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte - one pipeline for ingest, transform, and load that runs anywhere, and can also run dbt on DuckDB inside the same tool.

Three things set it apart:

1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs wherever Duckle does - no API key, no telemetry, no vendor round-trip. Point it at your own OpenAI-compatible endpoint instead if you would rather it did not run in-process.
2. 360+ components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
3. A self-contained binary you can audit. 73 to 110 MB depending on your platform. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

---

Why Duckle is different

---

Screenshots

Real pipelines, built and run in Duckle - not mockups.

_{A 5M-row pipeline: a CSV, a Parquet file, a DuckDB table, and a SQLite table enriched through one visual Map (3-way join), no SQL.}

_{Left: the visual Map editor - main plus lookups, per-output expressions, an inline filter. Right: Parallelize fanning out aggregate, window, and top-N branches.}

_{One run, many branches: 16 nodes finish in a few seconds. Concurrency auto-detects from CPU cores; branches write to Parquet, CSV, DuckDB, and SQLite at once.}

_{Left: DuckLake CDC change-feed mirrored via upsert + delete propagation (100k rows). Right: watermark incremental load over 5M rows, advancing state only on a fully successful run.}

---

Quickstart (60 seconds)

1. Download the binary for your OS (see Download / Install above) - or build from source.
2. Launch it. First run shows the setup modal:
   • Click Install on DuckDB (required, takes ~30 s).
   • Optionally click Install on Duckie AI Assistant (~1.1 GB, takes 5-10 min on average broadband).
3. Pick a workspace folder. Pipelines, connections, context variables, and routines live there as plain files.
4. Build a pipeline two ways:
   • Drag + wire: drag a CSV source in, point it at samples/orders.csv, hit Autodetect schema. Drag a Filter, wire it up. Drag a Parquet sink with an output path. Press Run, watch the nodes light up.
   • Ask Duckie: click the Sparkles icon (top-right of the toolbar), type "read orders.csv, filter where status = 'paid', write to paid.parquet". When Duckie streams back a pipeline, click Insert into canvas.
5. Inspect. Click any node to see its generated SQL in the Plan tab and a live row sample in the Preview tab.

That's a real, native ETL pipeline built and run in under a minute. CSV is just the easiest first node; swap in Parquet, JSON, S3, Snowflake, MongoDB, or Stripe the same way.

---

Download / Install

Pick the binary for your OS from the latest release:

The single-file binary above is all you need for Build Pipeline too: the headless runner is embedded into the app at build time, and exporting a pipeline produces ONE self-contained executable (the engine, the DuckDB CLI, any needed extensions, and the resolved pipeline are all inside that one file). Copy that single file to your server and run or schedule it - no separate runner download required.

One command, nothing installed: it scaffolds sample data and a pipeline, compiles it to SQL, runs it on DuckDB, and shows you the rows.

uvx duckle quickstart

Let an agent do it

Paste this into Claude Code, Cursor, or Codex:

Run uvx duckle quickstart to build my first pipeline and run it

Nothing to install first. The agent fetches Duckle and the DuckDB engine on demand, runs a real pipeline, and shows you the rows.

CLI only (CI, cron, containers)

If you do not want the desktop studio, install just the headless runner. It is about 27 MB rather than 100 MB or more, has no GUI dependency, and is what a build step actually needs.

pip install duckle

That is the whole install. It brings the DuckDB CLI with it (via the duckdb-cli package published by the DuckDB Foundation), so there is nothing else to fetch and it works offline. Wheels ship for Linux, macOS and Windows on x86-64 and arm64.

It also gives you a Python API, where pipelines are built as code and executed by DuckDB rather than by Python:

import duckle
from duckle import col

(duckle.read_csv("orders.csv")
    .where(col.amount >= 20)
    .derive(total="round(amount * 1.2, 2)")
    .write_parquet("out.parquet")
    .run())

Python expressions compile to vectorized SQL at plan time, so no rows pass through the interpreter. See the PyPI page for the full API.

The same package provides the duckle command-line runner for CI, cron, and containers - it bundles the headless runner and the MCP server per platform:

pip install duckle          # or run ad hoc, no install: uvx duckle --help

Pipelines execute as SQL on the DuckDB CLI, so the runner needs a duckdb on PATH or DUCKLE_DUCKDB_BIN set (pip install duckdb-cli is the quickest route). Validation does not:

duckle validate                 # compile-check every pipeline under ./pipelines
duckle validate --json          # machine-readable, for a CI step
duckle --pipeline my.json       # run one

validate opens no source and writes no sink, so it needs no engine, no credentials and no network. Exit codes are stable: 0 clean, 1 a real finding (a pipeline failed or did not compile), 2 the runner could not start (bad usage, unreadable file, missing engine).

The binary is 73 to 110 MB depending on platform (it embeds the headless runner and the bundled MCP server). On first launch you'll be guided through downloading two engines into your app-data directory:

App-data location:

• Windows: %APPDATA%\io.duckle.app\engines\
• macOS: ~/Library/Application Support/io.duckle.app/engines/
• Linux: ~/.config/io.duckle.app/engines/

Delete the engines/ folder if you ever want to force a fresh install.

---

Run your first pipeline

A worked example using the bundled samples/orders.csv data.

1. Add a source

• Open the Components sidebar (left). Click Sources -> Files -> CSV.
• Drag it onto the canvas.
• In the right-side Properties panel:
  • Path: browse to samples/orders.csv
  • Click Autodetect schema - the Schema tab fills in column types from the file, the Preview tab shows the first 20 rows.

2. Add a transform

• Components -> Transforms -> Rows -> Filter. Drag onto canvas.
• Wire the CSV source's main output port to the Filter's main input.
• In Properties:
  • Predicate: status = 'paid' (you can write raw SQL or use the visual builder)
  • Filter has two output ports: pass (rows matching) and reject (rows that don't).

3. Add a sink

• Components -> Sinks -> Files -> Parquet.
• Wire Filter's pass port to the Parquet sink.
• Path: paid_orders.parquet. Write mode: overwrite. Compression: zstd.

4. Run it

• Press Run in the toolbar. Nodes light up in execution order; row counts appear under each.
• Open the Output tab (bottom panel) to see per-stage timing.
• Click any node to inspect generated SQL in Plan + sampled rows in Preview.

5. Iterate

• Add a Group By before the sink to aggregate. Re-run. Sub-second on small data.
• Cancel mid-run with the Stop button - the DuckDB process is killed cleanly.
• Save your work: Cmd/Ctrl-S writes a JSON pipeline file to your workspace folder.

---

Where Duckle runs

You build a pipeline on your laptop. The server runs that same file. Nothing is rewritten, exported or converted in between.

flowchart LR
    D["Duckle Desktop<br/>your machine"] -->|deploy, needs admin| W
    B["Console in a browser<br/>your machine"] -->|turn it on, needs operator| W
    W["Workspace on your server<br/>a new schedule lands OFF"] --> C["Scheduler<br/>every 15s, takes what is due"]
    C --> R["It runs<br/>on that box, unattended"]
    R --> O["Run history, logs, metrics,<br/>alerts, and an audit log"]
    O -->|you watch it here| B

Nothing here depends on a person's machine being switched on:

• Pipelines are plain files in git. Review them in a pull request, roll them back, and let them outlive whoever wrote them. There is no proprietary repository and no exported binary artifact.
• The console has roles and an audit log, so more than one person can operate it and you can see who did what.
• Secrets are not in the pipeline file. They resolve from the environment or an encrypted per-workspace store at run time.

Working recipes for AWS (EC2, ECS, EKS), Azure (VM, Container Apps, AKS) and Google Cloud (Compute Engine, GKE), with manifests and the mistakes worth avoiding, are at duckle.org/deploy. Three things worth knowing before you start:

• On a non-loopback bind with no credential the console starts unclaimed, and for 15 minutes anyone who can reach it can claim it and become its administrator. Pass --token, set DUCKLE_CONSOLE_TOKEN, or create accounts with duckle-runner console add-user before exposing it. An empty value is refused rather than treated as absent, so an unresolved secret fails loudly instead of opening that window. Who can do what, and how one request is decided, is set out under Sign-in and roles.
• The scheduler runs in serve, not in the editor. Start the editor with schedules armed and it now says so rather than leaving you to wonder why nothing fired.
• GET /healthz needs no credential and answers ok, so a Kubernetes probe or a load balancer can check liveness without holding a token. Every other route is authenticated, so pointing a probe anywhere else reports the pod unhealthy forever.

Deploying a pipeline to a running server

POST /api/deploy lands a pipeline on a server from wherever it was authored, with the schedule it should eventually run on:

curl -X POST https://duckle.internal/api/deploy \
  -H "Authorization: Bearer $DUCKLE_TOKEN" \
  -d '{"name":"orders-load",
       "pipeline": '"$(cat orders-load.json)"',
       "schedule":{"intervalMinutes":30}}'

Two things are deliberate. The schedule arrives disabled, so a cadence someone set while testing on a laptop cannot start firing the moment it reaches production; enabling it is a separate call. And deploying needs admin while enabling needs operator, because a deployed pipeline runs shell and SQL on that host: shipping the code and starting it are two acts, and the audit log records both with the name of whoever did them.

API keys, for machines

A person signs in and gets a session. A machine has no browser and nobody to rotate a password, so it gets a key of its own:

duckle-runner console key-add ci-deployer --role admin --expires-days 90
duckle-runner console key-list      # role, state, and when each was last used
duckle-runner console key-revoke ci-deployer

A key carries its own role, so a deploy runner can be admin while a metrics scraper is viewer. It is printed once and stored only as a hash, so a lost key is replaced rather than recovered. key-list shows when each was last used, which is the question actually worth answering before revoking one, and revoking takes effect immediately on a console that is already running rather than at the next restart. Revoked keys are marked rather than deleted, so a key that turns up in an old log can still be named.

Accounts, sessions and keys live in .duckle/console.db. An existing console-users.json is carried into it on first start and renamed to .migrated, so an upgrade neither locks anyone out nor destroys the only copy of a credential store. For the roles, and a diagram of how one request is decided, see Sign-in and roles.

Scaling it

duckle-runner serve is an ordinary service. Run it on EC2, EKS, a VM or a container next to everything else you operate, and scale it the way you scale any service:

• More cores. The engine is parallel and uses every core on the box by default, so a bigger instance is a faster pipeline with no change to the pipeline. Bound it with DUCKLE_THREADS when you would rather it did not take the whole machine.
• More RAM. Set memoryLimitMb per stage, or a workspace default, and spill to disk past it.
• More pipelines at once. DUCKLE_MAX_CONCURRENT_RUNS raises how many run together; it ships at 1 so an unattended server stays predictable until you decide otherwise.
• More machines. duckle-runner work drains a queued batch from as many workers as you start, on as many hosts as you like, each claiming its items under a lock so nothing runs twice.
• Bigger than any box. Turn on pushdown and the query runs verbatim inside Postgres, Oracle, SQL Server or Snowflake. The warehouse does the scan; Duckle keeps the scheduling, lineage, data quality and alerting.

Measured, rather than asserted:

• 96,000,000 rows out of live Postgres to Parquet in 39.9s (details)
• Oracle extract at 65.0s, against 68.6s for python-oracledb with pyarrow on the same machine (details)

The one thing Duckle does not do is split a single query across a cluster the way a distributed warehouse does. When you need that, push the work down into the system that has it and let Duckle orchestrate around it.

Server deployment (Build Pipeline)

Want the studio to publish straight to a running server instead? That is the other route: connect a server once, then Deploy to a server from the editor. Step by step in docs/current/server-deployment.md.

Promoting from CI instead, or driving Duckle from Airflow, Dagster or Temporal? See docs/current/ci-and-orchestration.md, with copyable GitHub Actions and GitLab CI templates in docs/ci/.

Want to know exactly what crosses the wire, and where every credential is stored? docs/current/client-server-architecture.md is the diagrammed answer, sharp edges included.

The in-app scheduler runs only while Duckle is open. To run a pipeline on a server with no desktop app, Build Pipeline turns it into ONE self-contained executable - the equivalent of a standalone "Job".

Right-click a pipeline (in the project tree or on the canvas) and choose Build Pipeline. The output is a single file named after the pipeline (orders_etl.exe on Windows, orders_etl on macOS / Linux) that embeds everything it needs:

• the headless execution engine,
• the DuckDB CLI,
• only the DuckDB extensions that pipeline's components actually use,
• the resolved pipeline (context variables substituted, routines inlined),
• its secrets (see below).

On first run it self-extracts to a temp cache and uses its own embedded DuckDB, so the server needs nothing installed - no Duckle, no DuckDB. There is no folder to copy, no run.sh, and no separate runner download. A CSV-to-CSV pipeline builds to about 28 MB; only the extensions a pipeline uses are bundled, so the file stays lean.

./orders_etl            # or orders_etl.exe on Windows

The process exits 0 on success and non-zero on failure, and writes the same NDJSON run logs under logs/ (Splunk / Dynatrace friendly).

Build options

Schedule it with whatever the server already has - point the OS scheduler straight at the file:

# Linux cron - run every day at 02:00
0 2 * * * /opt/duckle/orders_etl >> /var/log/orders_etl.log 2>&1

On Windows use Task Scheduler; on macOS a launchd plist; on Linux a systemd timer. Full examples in docs/current/scheduler.md.

Run against an existing workspace - the same embedded headless runner can also execute a pipeline JSON directly, resolving context the way the app does:

duckle-runner --pipeline /path/to/pipeline.json [--workspace /path/to/workspace] [--duckdb /path/to/duckdb]

Continuous mode (follow)

A scheduled pipeline already consumes a stream without gaps: a source that tracks its position (src.kafka with trackOffset, xf.incremental) resumes where the last successful run stopped. What a schedule cannot give is latency - the scheduler wakes every 15 seconds, and every run pays process start, DuckDB resolution and document parsing again.

follow keeps the same execution model and removes that per-batch overhead. The document is read and resolved once, the engine is built once, and the pipeline then runs in a loop. Each pass is one micro-batch:

duckle-runner follow /path/to/pipeline.json --idle-ms 500

A failed batch never advances the source position. The position is queued during the run and written only when the run reaches ok, which is after every sink has written - so a failure anywhere, transform, quality gate or sink, leaves the position where it was and the next pass re-reads exactly the records that did not land. Killing the process is safe for the same reason; Ctrl-C finishes the batch in hand first, which only saves you a truncated output file.

That ordering is the difference between a correct micro-batch loop and a lossy one, so it is covered by a regression test that fails if the position ever advances past a batch that did not land.

Backfill without the desktop app (backfill)

Production deployments are headless, so replaying from an earlier point should not mean getting at the server's workspace through a GUI:

duckle-runner backfill list  --pipeline ./pipelines/daily.json
duckle-runner backfill set   --pipeline ./pipelines/daily.json --node inc --value 2026-01-01 --type TIMESTAMP
duckle-runner backfill clear --pipeline ./pipelines/daily.json --node inc
duckle-runner backfill list  --pipeline ./pipelines/daily.json --json      # for CI and agents

Five node kinds keep state in that folder, and only two resume from a value a person can write down. xf.incremental (a watermark) and src.ducklake.changes (a snapshot id) can be set; a src.kafka resume offset, a src.spool byte position and an xf.tumble buffer pointer are listed and can be cleared, but set on them is refused. Writing {value,type} over a tumbling window's state would drop the pointer to the rows it is holding and delete them on the next run, with nothing to report it.

Clearing is not always a full reload, and the tool says so: a Kafka node with startFrom: latest skips whatever is already in the topic when it has no saved offset, so clearing it moves PAST that backlog rather than replaying it.

The same three operations are on the console API and MCP, so a replay can be driven from CI or an agent as well as the CLI:

GET    /api/watermarks?file=pipelines/daily.json          viewer
POST   /api/watermarks?file=pipelines/daily.json          operator
DELETE /api/watermarks?file=pipelines/daily.json&node=ID  operator

Reading needs a viewer; changing what the next run processes needs an operator. All four surfaces - desktop panel, CLI, API, MCP - call the same engine functions, so the kind guard cannot be bypassed by picking a different one.

Push sources that do not lose what arrives (listen + src.spool)

src.webhook and src.websocket collect INSIDE a pipeline run: they bind or connect, take N messages or time out, and stop. Right for a one-shot capture, wrong for anything continuous - between runs the port is closed and arriving requests are refused. Under follow that gap is every batch boundary.

listen is the other half. It keeps the listener up and appends what arrives to an append-only NDJSON spool; a pipeline reads that spool with src.spool, from wherever the last successful run stopped:

duckle-runner listen --port 9000 --spool ./spool/hooks.ndjson --path-filter /hooks
duckle-runner follow ./pipelines/hooks.json --idle-ms 500

Arrival is decoupled from processing, so a slow batch, a failed batch or a restart costs nothing that already arrived. Append-only plus a byte offset is the whole trick: the reader never deletes and the writer never rewrites, so there is no race between them.

A record is {received_at, method, path, headers, json|body} - a JSON body is embedded under json so the pipeline can address its fields, and anything else is kept verbatim under body rather than dropped for not parsing. The spool is written and flushed BEFORE the 200 goes out, because a 200 tells the sender its delivery is safe and webhook senders do not retry those.

Resource budget (--memory-limit, --threads, --max-temp-size)

Duckle targets one machine and will use it. On a dedicated box that is what you want; on a shared server one unexpectedly large job should not be able to take everything else down with it.

duckle-runner --pipeline ./daily.json   --memory-limit 24GB --threads 8   --temp-dir /data/duckle-tmp --max-temp-size 300GB

--max-temp-size is the one worth setting deliberately. DuckDB's own default is 90% of available disk space, so without it a single large join or sort can fill the volume the OS is on - which is an outage, not a slow pipeline. --memory-limit is a spill threshold rather than a hard ceiling: above it DuckDB writes to the temp directory and keeps going, so the limit buys predictability, not failure.

Each run spills into its own subdirectory of --temp-dir. Pointing several concurrent runs at one shared directory is what a person does to move spill onto a bigger disk, and it used to make them unsafe: four concurrent spilling queries sharing a directory lost 3 of 12 to segfaults and delete failures.

The flags set the same variables the engine reads (DUCKLE_MEMORY_LIMIT, DUCKLE_THREADS, DUCKLE_TEMP_DIR, DUCKLE_MAX_TEMP_DIR_SIZE), so a flag, a workspace-wide export and a per-stage setting all land in one place, with the most specific winning.

Poll a remote source without downloading it (src.changed)

A pipeline that watches a bulk source should not pay for the object to find out whether it was needed. src.changed compares what a HEAD or an SFTP stat reports against the last fingerprint it successfully processed, and emits a row only for what moved. https://, s3:// (including MinIO, Backblaze B2, Cloudflare R2 and other S3-compatible stores, through a saved connection or credentials on the node) and sftp://.

Two shapes, because they are the same question asked of a different number of objects:

• object - one URI replaced periodically. A row when its fingerprint differs, nothing when it does not.
• listing - an sftp:// directory or an s3:// prefix of immutable files. Lists it, compares each entry, and emits the new and changed ones as ordinary rows for a ctl.foreach or an artifact copy downstream. S3 listings follow continuation tokens, so a prefix larger than one page is enumerated fully rather than silently truncated at the first thousand.

Rows carry uri, name, size, modified_at, etag, fingerprint and status (new / changed).

A quiet poll is not a plain success. When nothing changed the node reports unchanged, so a working poll and a broken one are told apart - a healthy source can be unchanged hundreds of times between updates, and that has to stay countable.

Fingerprints are conservative on purpose. None of the signals are guarantees: an ETag can be absent, can weaken under compression, and on S3 is a digest-of-digests for a multipart upload rather than the object's hash; Last-Modified has one-second resolution; SFTP offers mtime and size. A missing or unreadable signal therefore counts as changed. Re-reading something unnecessarily costs compute; skipping something that did change loses data and reports nothing.

What was processed advances only when the whole run succeeds, and only for rows that were actually emitted - so a failure downstream re-offers the same files, and a run capped by maxEntries does not mark the remainder as done.

Maintain a DuckLake through the same pipelines (src.ducklake.maintain)

A lakehouse that is written to continuously eventually needs maintaining as well as filling: frequent incremental writes leave many small files, snapshots accumulate, and files stay referenced longer than they need to be. Those operations used to live outside Duckle.

Each operation is one DuckLake function, and its options are that function's options - compact, rewrite files heavy with deletes, expire snapshots, clean up files an expired snapshot released, delete orphaned files, flush inlined data, or read per-table storage statistics. Nothing here invents storage semantics, so what it does follows the installed DuckLake rather than anything Duckle decided.

The result comes back as ordinary rows, which is what lets a quality check or an alert read a compaction the way it reads anything else, and the node reports what changed: ducklake compact: 1 row(s) - files 4 -> 1, 1.1 KB -> 513 B.

Three things about deleting, since that is where this gets dangerous:

• The three destructive operations support dry run, which lists exactly what would go and changes nothing.
• Ticking dry run on an operation DuckLake cannot dry-run is refused, not ignored - an ignored dry run deletes while the operator believes nothing will happen.
• Snapshot expiry does nothing without an explicit retention boundary. That is DuckLake's own default and it is surfaced rather than replaced, so a scheduled job that forgot its boundary does nothing instead of deleting history.

Two maintenance runs against one catalog serialise on a lock rather than racing, so a weekly compaction overlapping a monthly cleanup waits instead of failing a two-hour job at its commit.

Land the bytes somewhere durable (xf.artifact.copy)

An artifact is a reference - a uri, a media type, a size, a hash - so a pipeline can carry one around for nothing. At some point the actual bytes have to move, and that is this step: between "the feed says there is a new 4GB bundle" and "it is in our raw zone, hashed, and we can prove which bytes we parsed".

It reads a uri column - whatever src.changed, src.artifact or a query produced - and copies from https://, s3://, sftp:// or a local path to an s3:// prefix or a local directory.

Streamed and hashed in one pass. Memory is bounded by the part size rather than by the object, so a 40GB model file does not become 40GB of RSS, and the sha256 recorded is of the bytes that actually transferred. Reading twice - once to hash, once to upload - would double the transfer off a remote source; hashing first would mean holding the whole thing.

Naming is keep (the source's file name), path (its layout preserved under the prefix) or hash (content-addressed, which makes the store immutable and de-duplicating at the cost of reading each source twice, because the key is the hash). A source-derived name can never climb out of the destination prefix.

ifExists: skip is the default and is what a raw zone wants: re-running a feed does not re-upload what already landed. The row still comes out, with copied = false, because downstream still needs to know the artifact exists.

Emits uri, source_uri, name, media_type, size_bytes, sha256 and copied.

Remote artifacts reach the signed run manifest. .ducklock pinned local file inputs by path, and a remote object has no path - so the boundary that matters most in a raw-zone pipeline, where the bytes came from, was the one thing the manifest did not record. Every object a run reads or writes now appears in it with its uri, size, media type, and either a sha256 when the bytes actually passed through the run or an ETag and mtime when they did not. An object that was merely observed carries no hash, because claiming one would be a lie. The manifest also records the resource limits the run was given, so two runs that spilled differently can be told apart from two runs handed different budgets.

Tumbling windows that survive between runs (xf.tumble)

Aggregating a stream by time needs a window to stay open across batches, and needs to know when it can be closed. xf.tumble assigns each row to a fixed-size bucket by its EVENT time, holds it until the bucket closes, then emits it with window_start / window_end for an ordinary GROUP BY downstream.

Closing is decided by a watermark - the greatest event time seen so far, across runs - not by the wall clock. Replaying last year's data therefore produces last year's windows, instead of finding every one of them older than "now" and closing the lot at once.

allowedLateness holds a window open past its end for out-of-order arrivals. Anything that arrives after its window was already delivered is dropped and counted, not emitted: sending it would hand a downstream consumer a second, partial copy of a window it already has, with different numbers in it.

The rows in still-open windows and the watermark ride the same deferred flush as every source position, so a batch that fails downstream leaves them intact and re-processes rather than losing what it was holding.

Web panel (remote management console)

To run and monitor pipelines on a server with a browser instead of the desktop app, start the built-in web panel - it is part of the same duckle-runner binary, so there is nothing extra to install:

duckle-runner serve --port 8080 --workspace /path/to/workspace

Open http://localhost:8080. The panel has eight views:

• Overview - every pipeline with its last status, duration and next scheduled run, and a Run button.
• Runs - run history across every pipeline (status, duration, rows, errors) with expandable per-pipeline run logs and optional auto-refresh.
• Schedules - an editable cron or interval schedule per pipeline, showing what is running now and what is due next.
• Plans - several pipelines in the order you chose. See Plans.
• Catalog - everything the workspace reads and writes, who owns it, and what is written but never read. See Workspace catalog.
• Batches - work queued for workers: progress, what is running now, what failed, and a retry for the failures.
• People - the accounts that may sign in and the keys machines use, with the role each one has. Admin only.
• Audit - who signed in, what they changed and who was turned away. Admin only, and shown only to admins.

Runs execute in-process through the same engine, are written to the same run history (<workspace>/runs/) and logs (<workspace>/logs/), and a built-in scheduler triggers any pipeline whose schedule has elapsed - so the server itself runs your schedules, no OS cron needed.

Plans: several pipelines, in an order you chose

A schedule runs one pipeline. A plan runs several, in steps: everything inside a step goes at once, and the next step waits for it. A step that fails stops the ones after it, so nothing runs against data that was never produced.

That is the shape most nightly loads already have. Without it they get written as three schedules set a few minutes apart and hoped over, which works until the extract takes four minutes instead of two.

Build one wherever you are: the Plans tab in the web console, or the Plans tile under Operate in the desktop app. Add a step, put pipelines in it, and the card draws the chain it will run.

EXTRACT                    PUBLISH
orders.json      -->       export.json
customers.json

Two things worth knowing:

• Every pipeline keeps its own run history. A plan does not collapse into one opaque run, because at three in the morning the question is which step broke, not that the nightly load did.
• A plan can be scheduled like anything else, from its own card. The same plan runs whether the schedule is fired by serve on your server or by the desktop app on a shared workspace - both read plans.json and schedules.json, and both decide it the same way.
• It is one file, so it travels. A plan written in the desktop app opens in the console and the other way round, and it goes to your server with everything else in the workspace.

Plans live in <workspace>/plans.json, so they are a file in git alongside the pipelines they order.

Sign-in and roles

Start from where you actually are.

Running it on your own machine? Nothing to do. On 127.0.0.1 with no accounts the console is open, because anyone who can reach it is already sitting at the machine, and asking them for a password would protect against an attacker who has already won.

Put it on a server and it refused to start? That is the feature. The console can run any pipeline in the workspace, and a pipeline can run shell and SQL, so reaching it is the same as running code on that host. A bind it cannot authenticate fails rather than serving anyone and printing a warning nobody reads. The shortest way past it:

DUCKLE_CONSOLE_TOKEN=<secret> duckle-runner serve --host 0.0.0.0 --port 8080

More than one person? Give each of them their own, so the audit log can name them. The token is printed once and kept only as an Argon2id hash:

duckle-runner console add-user reporting --role viewer
duckle-runner console add-user ops       --role operator
duckle-runner console list

A machine needs in? CI, a scraper, or your own laptop deploying: those have no browser and nobody to rotate a password, so they get a key instead of an account. See API keys.

The split follows what an action can destroy, not which screen it lives on. It is why deploying a pipeline needs admin while turning its schedule on needs operator: shipping code to a host and deciding when trusted code runs are different sizes of decision.

How a request is decided

Three ways to prove who you are, one identity, one check, and every outcome recorded:

flowchart LR
    R([Request]) --> C{"Session cookie?"}
    C -->|within 12h| ID["Identity<br/>name + role"]
    C -->|no| B{"Bearer token?"}
    B -->|API key| ID
    B -->|account token| ID
    B -->|nothing| U["401<br/>sign in"]
    ID --> P{"Role enough<br/>for this route?"}
    P -->|yes| OK["It happens"]
    P -->|no| F["403<br/>refused"]
    OK --> A[("audit log<br/>who, what, when")]
    F --> A
    U --> A

Two things follow from that shape. A refusal is recorded as carefully as a success, so audit --outcome denied answers "who is reaching for what they do not have". And a route with no entry in the permission table needs admin, so a route added later is locked down rather than accidentally left open.

A browser trades its credential for a session cookie, so it never stores the credential itself: the cookie carries a random session id, is HttpOnly and SameSite=Strict, is marked Secure when a proxy tells Duckle the browser is on https, and lasts 12 hours. Sessions survive a restart, so a rolling deploy does not sign your team out.

Accounts, sessions and keys live in <workspace>/.duckle/console.db. Nothing in it can be replayed: an account token is an Argon2id hash, and a session id and an API key are both generated with 256 bits of entropy and stored as SHA-256, so a copy of the file or a backup of the workspace admits nobody. An older console-users.json is carried in on first start and renamed .migrated, so upgrading neither locks anyone out nor destroys the only copy of a credential store. The same accounts, roles and keys cover duckle-runner web.

Read it back from the Audit view, or from a terminal with no server running:

duckle-runner audit                                  # newest first, 50 by default
duckle-runner audit --outcome denied                 # who reached for what they do not have
duckle-runner audit --actor ops --action schedule    # one person, one family of actions
duckle-runner audit --limit 500 --json               # for a collector

allowed means the caller was permitted to proceed, not that the work then succeeded - run history answers that. Reads are not recorded, so a dashboard polling every few seconds does not bury the entries worth seeing. A page says when older entries exist beyond it, and a line that will not parse is counted rather than silently skipped.

Still put it behind a reverse proxy if you need TLS.

Migrating a repository of legacy jobs

The editor imports one job at a time, which is how you try Duckle. This is how you leave another tool: point it at a checkout and convert everything.

duckle-runner import ./legacy-jobs                    # convert the tree into ./imported
duckle-runner import ./legacy-jobs --out ./pipelines  # somewhere else
duckle-runner import ./legacy-jobs --json             # for a migration script
duckle-runner import ./legacy-jobs --strict           # CI gate, exits 1 if anything needs a person

The folder layout is mirrored rather than flattened, because two jobs in different folders routinely share a name and flattening would silently drop one. Files that are not jobs - routines, contexts, SQL templates - are reported separately rather than counted as conversions, and no empty pipeline is written for them.

Reusable job bodies convert alongside jobs, and a job's children resolve to the files they became, so the master/child/joblet graph survives the move rather than arriving as a set of disconnected pipelines. A loop or iterate body is lifted into its own pipeline that the parent calls, which is why the file count comes out higher than the job count.

The closing tally is the number to decide on. It says how many jobs came across clean, how many need a person, and which components have no equivalent yet, sorted by how often they appear. On a real 125-file corpus that list had a single entry, a site-specific custom component: coverage is the head of the distribution, so a corpus usually converts far better than a raw component count suggests.

What remains is credentials and Java. Credentials were never in the job files: encrypted passwords become ${ENV:...} placeholders and connections defined outside the job are named so you can point them at a saved connection. Java is the part that needs a person, and the report separates it so you can see how much there is:

• A mapper expression is translated when it has one faithful SQL reading: a literal, a cast, a character function, a comparison, a choice, arithmetic. Anything whose meaning depends on a Java type the job file does not record stays reported, because guessing there produces a silently wrong number rather than a failure.
• A Java body is never turned into something that compiles. It imports with no SQL and fails, since a pipeline that runs while omitting the rules is worse than one that stops. A body whose every statement is a print carries no rules and is called out separately, so a long list sorts into what can be deleted and what has to be ported.
• A body whose every statement sets a context value is carried over instead, as one Set Run Variable node per value, wired in the order the body set them. All of it or none of it: one statement that cannot be read leaves the whole body for a person, because carrying half of it over leaves something that looks finished and is not. A body that took its values from the row it was given ran once per row, so the last row decided what they held, while a node sets them once from the first row - the same thing for a single row, which is what these bodies are usually fed, and the report says which nodes it applies to rather than leaving you to notice.

A component with no equivalent is imported as a named placeholder and reported. That includes a job body's input and output ports: a child pipeline runs for its side effects, so it does not yet take rows from its caller or hand them back.

A SQL step that changes the database is reported, not converted. A SQL step returns rows and compiles into a view, so a step carrying an UPDATE, MERGE or CREATE cannot become one: it would reach the database wrapped in CREATE VIEW and fail there. That is knowable at import, so it is said at import. On a 125-file corpus, 16 steps.

How a write writes is carried across. A warehouse sink records whether it appends rows or amends the ones already there, and importing that as the default write mode turned an append into a full-table replace - so on a table several nodes write to, each one erased the one before it. The write action now comes across, with the key it matches on taken from the columns the schema marks as keys. An action with no exact equivalent here is reported rather than widened in silence.

The order subjobs run in is kept. Most subjobs are not linked to each other at all; they run one after another in the order the file lists them at the end, and that order was being dropped. A job that wrote a table in one subjob and read it in the next then arrived as two things that could happen in either order. Branches of a parallel fork are the one part that genuinely does not run in declared order, so they are left out of the chain.

Intermediate work moves to DuckDB. A job written against a warehouse uses it as working storage as well as a destination: it writes a staging table, reads it back, joins it, writes it again, and every one of those hops is billed for rows that were produced on this machine in the first place. So a table the imported project both writes and reads is mirrored into <workspace>/.duckle/staging.duckdb as it is written, and the reads are pointed at the mirror. The warehouse write is left exactly as it was, which is what makes this safe to do unasked: every table still lands where it landed before, so nothing downstream of the project can tell the difference. Only the reads move.

A read moves only when the whole of it can. A query that also names a table the project does not write still needs the warehouse to resolve it, so that read stays - and so does the staging table it reads, since a mirror would then be serving only half of what the project asks for. Neither does a read the job could run before its own write: within one pipeline the write has to lead to the read, by rows or by an ordering link, because a warehouse table nothing wrote yet holds stale rows while a local one is simply not there. A mapper's second input is judged by where its mapper sits, since that is when a lookup is loaded and nothing feeds the lookup itself; where such a read does move it is held until the mirror has been filled. A mapper that reads a table and produces the write back to it keeps reading the warehouse, because what the lookup feeds is what changes the table. Anything else, including a query assembled at run time, is mapped as it was.

Workspace catalog (what reads and writes what)

Every other lineage view in Duckle answers about one pipeline. The catalog answers about the whole workspace, by joining pipelines through the assets they name: two pipelines that read and write the same table are connected whether or not anyone drew a line between them.

duckle-runner catalog lint                                        # CI gate, exits 1 on findings
duckle-runner catalog diff main                                   # what this branch does to the graph
duckle-runner catalog build                                       # scan every pipeline
duckle-runner catalog assets                                      # every table, file, topic and endpoint
duckle-runner catalog impact postgres://db:5432/sales.public.orders
duckle-runner catalog orphans                                     # written here, read by nobody
duckle-runner catalog owners                                      # what nobody has claimed

impact is the blast radius: the pipelines that read an asset, the assets they write, everything downstream of those, and how many hops away each is. Assets that could not be named are counted on every answer rather than dropped, so a partial graph never looks complete. An asset name nothing in the workspace uses exits non-zero, including under --json, so a mistyped name in a CI gate fails instead of reporting an empty blast radius.

build walks the whole workspace, skipping Duckle's own folders (runs, logs, connections, .duckle), so pipelines kept in subfolders are included. The saved graph records what it was built from, so it knows when the pipelines have moved on: the CLI rebuilds on read rather than answering from a stale graph, and the console says "pipelines have changed since this was built" instead of quietly presenting an old blast radius as current. The console reports it rather than rebuilding, because reading the catalog is a viewer action and rebuilding writes a file - Rescan is the operator's button. The check is stat only, so it costs nothing to make on every read; a same-length edit inside the same millisecond would slip through, which is the price of not hashing every pipeline on every read. Asset names never carry a credential: a mongodb://user:pass@host uri or an ODBC connection string is reduced to the address before it becomes a name, which also keeps the name stable when the password is rotated.

Add <workspace>/owners.json and it also tells you who to notify. Rules are globs and the first match wins, so a narrow rule above a broad one carves out an exception:

{
  "assets": [
    { "match": "/lake/raw/pii_*", "owner": "Privacy Office", "contact": "privacy@example.com",
      "description": "Landing zone for regulated source tables.", "tags": ["raw", "pii"] },
    { "match": "/lake/raw/*",     "owner": "Data Platform",  "contact": "data@example.com" }
  ],
  "pipelines": [{ "match": "*-ingest-*", "owner": "Ingest Squad" }],
  "terms": { "active customer": "Ordered in the last 90 days." }
}

The same file carries the human half of the catalog: an optional description and tags per rule, and a workspace terms glossary for the words three teams would otherwise each define differently. Every one of those is optional, so an owners.json written before they existed still loads unchanged. They live here rather than in a file of their own because they are authored, reviewed and committed alongside ownership, and a second file would drift from this one.

Every run records which assets it read and wrote, with row counts, under the same names the graph uses - so the catalog can answer the first question anyone actually asks of an entry: is this current? A table nobody has written for three weeks is the interesting one, and no amount of structure reveals that. Only successful runs count towards freshness: a failed run may have written nothing, or half of something, and showing either as the last write would make a broken load look like a fresh table.

Assets also carry the columns the pipelines declare, unioned across every node that touches them - a pipeline reading three columns of a table another writes twenty to does not make the table three columns wide. They come from the schema a node already carries, so building the graph still opens no source and needs no credentials. No declared columns means none are known, which the catalog does not confuse with the asset having none.

In the desktop app this is the Data Catalog screen (Home -> Govern): search every asset by name, owner, tag or column; see who writes it, who reads it, what columns are declared and when it was last written; and set the owner, description and tags without leaving the app. Saving writes a rule for that exact name above any wildcard covering it, so describing one file never re-describes its neighbours. Read live schema opens the source on demand through a node that already reads it, so it authenticates the way the pipeline does - it is never done just because a screen was opened.

catalog lint is the gate for a CI job: it exits 1 when it finds something. It reports ownership rules that match nothing - almost always a typo or a renamed asset, and a failure that is otherwise silent, because the team the rule names simply never gets told about anything - patterns that will not compile (they own nothing, safely and invisibly), and nodes the graph could not name. Unowned assets are reported but only fail under --strict: most workspaces have a long tail nobody will ever claim, and failing CI over it on day one is how a useful check gets deleted from the pipeline instead of acted on.

catalog diff <rev> answers what a change does to the graph, which is the question a review of a data platform actually asks: which assets appear, which disappear, and - the one that matters most - which are still there but have lost every pipeline that wrote them. A deleted asset is loud, because something errors. An asset nothing writes any more is silent: no error, no missing file, the table simply stops moving and whoever reads it finds out weeks later. The revision is read straight from git's object store, so nothing is checked out and it is safe to run on a dirty worktree.

The console's Catalog view now shows the same facts as the desktop screen - description, tags, columns and freshness - because both are assembled by one function in the engine rather than two that would drift. The same answers are available over MCP as workspace_impact.

Alerting (tell someone when a run fails)

snk.email and snk.rest are pipeline nodes - they need wiring into every pipeline and cannot fire when a pipeline dies before reaching them. <workspace>/alerts.json watches the runs themselves, for both the desktop scheduler and the server:

{
  "rules": [
    { "match": "nightly-*", "channel": "webhook", "url": "${ENV:SLACK_WEBHOOK}", "cooldownMinutes": 15 },
    { "match": "*", "channel": "email", "smtpHost": "smtp.example.com",
      "from": "duckle@example.com", "to": ["oncall@example.com"] }
  ]
}

The webhook payload carries a text field as well as structured fields, so Slack, Teams and Discord render it directly. Three behaviours are deliberate:

• Repeat suppression. A five-minute schedule that starts failing would otherwise send 288 messages a day. cooldownMinutes (default 15) bounds it per pipeline and event.
• The all-clear. A success after a failure is its own recovery event and ignores the cooldown, so nobody is left thinking an outage is still running. Ordinary successes are silent unless you add "on": ["success"].
• It never breaks a run. Delivery happens after the run is recorded, is time-bounded, and an unreachable channel is logged rather than raised.

A schedule whose pipeline file has been renamed or deleted also raises an alert, instead of silently doing nothing.

---

Schedules and triggers

Pipelines can run on cron, fixed interval, or file-watch triggers. Configure these in the Schedule panel (toolbar -> Schedule icon), not as graph nodes.

Schedules persist to workspace/schedules.json and execute via the in-process scheduler crate. They survive app restarts but require Duckle to be running.

For headless / always-on schedules that run when Duckle is closed, build the pipeline into a standalone file and let the operating system's own scheduler run it - see Server deployment below.

---

Meet Duckie - the local AI pipeline assistant

Describe what you need. Duckie writes the pipeline.

The sidebar on the right is Duckie AI Assistant - powered by Qwen 2.5 Coder 1.5B running through llama.cpp, downloaded once (~1.1 GB) and then run entirely on your CPU. Ask in plain English; Duckie streams back a valid Duckle pipeline definition. One click drops it onto the canvas, ready to inspect, tweak, and run.

---

Benchmark

20M-row CSV into DuckDB

The most common job in data engineering: load a 20M-row CSV into DuckDB. One identical 2.49 GB file (20M rows of TPC-H lineitem, 16 typed columns), every tool measured at its best configuration, wall-clock time to land the data as a table.

How it was measured

• Machine: Intel Core i7-13650HX (14C / 20T), 24 GB RAM, NVMe SSD, Windows 11, DuckDB 1.5.4. Duckle, dlt and ingestr ran here.
• Best config per tool: dlt used the Arrow plus parquet loader path; Talend and Informatica used their bulk output connectors at max config (their default row-by-row sinks were 5-7x slower). Nothing was left on a slow default to pad the gap.
• Talend and Informatica ran on a separate 8 GB VPS, so per-tool peak RAM was not captured for those two.
• Airbyte (source-file to destination-duckdb) is scaled from measured 2M and 5M runs at a steady ~18k rows/s, and it also needs an always-on ~8 GB platform just to start.
• Wall-clock time, peak working-set of the whole process tree.

Why Duckle is this fast: its 15.69s sits right on top of raw DuckDB's own load floor (~16s to fully parse and write all 20M typed rows into an on-disk table). Duckle wraps the engine with pipelines, connectors, and a UI, then gets out of its way. That is the entire design goal. A read-only scan or aggregate over the same CSV is far faster still; this benchmark measures the heavier "materialize it as a table" job that every ETL tool here performs.

96M rows, Postgres to Parquet

A second, harder job against a live database: full-refresh extract of 95,988,640 rows of TPC-H lineitem (14 GB in Postgres 16) out to Parquet.

Run it yourself. The harness is in this repo at benchmarks/pg-to-parquet. ./bench.sh all brings up Postgres, generates the data at any scale factor, and times every tool you have installed. No timing is recorded until the output has been reopened and checked for the right row count and the right sum(l_orderkey), so a tool that writes a fast but wrong file gets a failure rather than a number.

Read it with these caveats

• The DuckDB floor is not a competitor. It is raw postgres_scanner plus COPY TO: no scheduling, no typing, no incremental state, no UI. It is there to show how much of the clock is the machine reading Postgres. Duckle landing 11% under it is the honest framing, not "Duckle beats DuckDB".
• ingestr has no Parquet destination and writes a DuckDB file, so its output size is not like-for-like. Its time is.
• Compression is not normalised. Duckle wrote zstd, the others snappy.
• Airbyte and Meltano are absent. Airbyte has no local Parquet destination and has only run against an earlier synthetic dataset; Meltano was not wired up. Neither is claimed here.

Hardware, per-run numbers and the two measurement traps that produced wrong figures on the first attempt are written up in RESULTS.md.

---

Status

Duckle is in public beta. The visual designer, the DuckDB execution engine, the scheduler, the cloud connectors, and the Duckie AI assistant all work today and are covered by 170+ integration tests across Linux, macOS, and Windows. The catalog is still growing and APIs may evolve before 1.0, but the day-to-day surface is stable enough for real work.

Scope, stated plainly: Duckle runs as a service on hardware you provision, and uses all of it. What it does not do is split one query across a cluster, so when a job outgrows the largest instance you want to pay for, push the work down into the source system or point the output at a warehouse, object store or lakehouse. It will not pretend to be a cluster.

The component palette ships 384 nodes so the roadmap is visible in the product itself:

• 366 available runs on the DuckDB engine today
• 3 preview is configurable in the designer (drag, wire, set properties); execution is being wired engine-by-engine
• 15 planned is reserved in the palette but not yet executable - see docs/roadmap.md

---

Capabilities

Duckle is not a CSV tool with extras. It reads a broad set of formats and sources, ships a deep transform library, and writes to files, databases, object storage, vector DBs, message buses, and email.

Sources

113 sources available today.

For CSV / TSV sources, the Schema panel accepts an optional per-column Format (a strptime token string such as %d/%m/%Y) on Date and Timestamp columns. Several date columns can each parse a different layout in one read - the column is read as text and re-parsed with its own format, working around DuckDB's single global date format. A value that does not match its format becomes null rather than failing the run. Set a Date or Timestamp column's Format to excel to convert Excel day-serials correctly. CSV sources also surface ignoreErrors (skip unparseable rows) and nullPadding (pad short rows with nulls) toggles in the GUI.

For JSON sources, a Format selector picks how the file is read (auto / array / JSON Lines / object), and a skip malformed records toggle drops records that fail to parse instead of failing the run.

Transforms

130 transforms available today.

All 6 AI transforms ship today. Three need a model API (LLM, Classify, Embeddings) and ride the apiKey-in-props pattern; three are pure-local (Chunk, PII Redact, Dedupe).

Data quality

27 validators available today.

Validators split their input: passing rows continue on the main port, failures route to a reject port you can sink, count, or inspect.

Custom code

7 ways to drop into code available today.

Sinks

73 sinks available today.

Database sinks support an optional dead-letter (validate-before-insert) step: rows that do not match the declared column types are split off to a dead-letter file (parquet / csv / json) and only the clean rows are inserted.

Control flow

18 control-flow components available today.

| Try / Catch | Install a fallback sub-pipeline if the wrapped stage fails | | Retry | Per-stage retry policy (configure on Advanced tab) | | Log Message | Emit an info log line ({rows} = upstream count), pass rows through (ctl.log) | | Warn | Emit a warning log line, pass rows through (ctl.warn) | | Die / Fail | Stop the run with a message: always, only when the input has rows, or only when empty (ctl.die) | | Schedule | Cron / interval / file-watch triggers via the orchestration crate |

A run variable is read as a value wherever the SQL of a later step names it: on its own, as a whole string literal ('${name}', the usual way to write a value into a WHERE clause, where the quotes come off with it), or inside a longer literal, which is joined around it. A name a node sets this way is left for the run to fill in, so a static context entry of the same name does not pre-empt it.

It also travels into whatever the pipeline runs. A Run Job, an Iterate or a For Each started after the value was worked out hands it to the child as ${name}, and on again to whatever that child runs, so a value settled in a master job reaches a body several levels down. A value named on the call itself still wins, since naming one there is how a parent says which value to run the child with. A name whose value came out NULL is not passed: it has no value, and an unset ${...} is left as it is rather than arriving as the word NULL.

A sub-pipeline runs under its own name, so its run log lands in logs/<child>/ and an xf.incremental watermark inside it is saved to state/<child>/<node>.json. Two different children driven by the same For Each therefore keep separate marks.

Set For Each -> Item key column to separate the ITERATIONS too. The child then runs as <child>@<value>, so loading 400 tables through one sub-pipeline keeps 400 watermarks in state/<child>@<table>/<node>.json instead of one. Leave it blank and every row shares a single mark, which silently skips rows when each row is a different table. It is never inferred from the row's position, because that would move every watermark the moment the driving query is reordered.

Set For Each -> Dispatch to Queue for workers and the rows are written to batches/<id>.ndjson instead of being run, one JSON line per row carrying the child reference and that row's substitutions. Nothing runs until a worker picks the batch up, so the run that queued it reports how many items are waiting rather than pretending they loaded. A batch is a file in the workspace like everything else here: no queue server, no database, no network service.

Queueing also reports whether the items are actually safe to spread out. Both "400 items each loading their own table" and "400 items appending to one file" look identical on the canvas - one sink node with a variable in the path - and only the first survives being run at once. So each item's variables are put into the child and the resulting targets are named with the same function that builds the workspace catalog, before anything picks the batch up:

duckle: 400 item(s) write to targets nothing else in the batch writes, so they
        are safe to run at the same time
duckle: heads up - 1 target(s) are written by more than one item (400 items
        write /lake/everything.parquet). Workers run items at the same time, so
        these will collide unless the sink is an upsert or the target is
        append-safe

It warns rather than refuses: appending many items into one table is a real thing to want, and only you know whether that sink is safe for it. Items whose child cannot be read are counted and reported, so a partial check never reads as a clean one.

Then run workers against it:

duckle-runner work --workspace /path/to/workspace     # drain every batch
duckle-runner work --batch fe-20260816T101112123      # just this one
duckle-runner work --once                             # one item, then exit

duckle-runner work status                             # what is stuck, and why
duckle-runner work retry --dead                       # start the stuck ones over

Start it on several machines pointed at one workspace and they share the batch. Each item is claimed with the same OS lock a pipeline run uses, so no two workers take the same one, and a worker that is killed mid-item leaves nothing to clean up: the kernel drops the lock and the item becomes claimable again. There is no lease, no heartbeat and no timeout, because there is nothing to expire. Progress is appended to batches/<id>.ledger.ndjson, so re-running a worker resumes rather than repeats.

Retries are bounded. A failed item stays claimable and is tried again on a later pass, which is right for a timeout and wrong for a 404 that will always be a 404: without a limit that item takes a worker slot on every pass forever. Set Max attempts per item on the For Each node, with a fixed or exponential backoff, and an item that uses them up is left alone and reported as dead rather than chased. work status lists what is waiting out a backoff and what is dead, with the last error; work retry --dead starts the dead ones over. A retry appends a reset marker rather than rewriting the ledger, so the failures stay readable - an item that died four times before someone fixed the source still says so. Leave max attempts at 0 and behaviour is exactly what it was.

Items run at least once, not exactly once. The ledger is written after an item succeeds, so a worker that finishes an item and then dies leaves it looking undone and another worker repeats it. That is the honest trade for having no transactional store - the alternative loses items instead of repeating them, and a lost load is worse. Make the child idempotent (an upsert sink rather than an append) and a repeat costs time, not correctness. A failed item stays claimable and is retried on a later pass, with the failure kept in the ledger.

The console has a Batches view: progress per batch, how many items are running right now, how many failed, and the recent attempts with the worker that ran each one. "Running" is answered by asking the run lock rather than by trusting a heartbeat, so a worker that died is not counted as running and there is no lease that could have gone stale. Retry failed clears the recorded failures so those items are claimable again, keeping the successes so a retry never repeats finished work.

Before running anything, a worker proves the lock actually excludes on that filesystem: it takes a lock and asks a second process whether it can take the same one. Some shared filesystems tell every caller it has the lock - NFS with no lock daemon is the classic case - and on one of those every worker would claim every item and each item would run once per worker, silently, with no error anywhere. A worker refuses to start there. Check it yourself with duckle-runner work --check; --no-check overrides, knowing the above. A test that could not be run is only a warning, because failing to prove exclusion is not the same as having disproved it.

Measured on one machine: three workers against a twelve-item batch took four items each, with no item run twice. Several machines against one shared filesystem is the design intent and is not yet measured, so treat it as untested until it is. scripts/measure-multi-host-batch.sh is the measurement: point it at a shared workspace and two or more hosts and it counts duplicate executions, failing if there are any.

Advanced settings (per-node)

Every node has an Advanced tab with fields the engine honours at run time:

Orchestration and workspace

---

Clean data before it reaches your AI

Models inherit the quality of their inputs. RAG indexes, embedding stores, and training sets quietly accumulate duplicates, nulls, malformed rows, mixed encodings, and inconsistent schemas. Duckle is built to scrub that data before it lands in a vector store:

• Deduplicate with exact Distinct, Uniqueness, and Fuzzy Deduplicate (Jaro-Winkler / Levenshtein); use Record Match to find near-duplicate pairs with a similarity score
• Semantic dedupe with xf.ai.dedupe over a precomputed embedding column
• Profile + describe every column up front (Column Profile, Describe, Histogram) so issues surface before they reach a model
• Validate and filter malformed, empty, or out-of-range records and route failures to a reject port
• Normalize types, encodings, casing, and null handling across messy sources (Standardize, Cast, regex / string transforms)
• Redact PII (emails, phones, SSNs, credit cards) via xf.ai.pii before embedding
• Chunk + embed long text via xf.ai.chunk -> xf.ai.embed for RAG indexing
• Classify rows with an LLM (xf.ai.classify constrains the model to one of N user-supplied categories)
• Retrieve with both halves of hybrid search, locally, no model API required: Vector Similarity Search (cosine / L2 / inner product) and Full-Text Search (BM25)
• Land it in your store - pgvector ships, and Pinecone, Qdrant, Weaviate, Milvus all have working sinks that POST batches through each vendor's HTTP API

---

Engines

Duckle ships a thin shell and installs its engines on first launch.

First-launch extension pre-fetch

When the installer downloads the DuckDB CLI it also pre-fetches the extensions Duckle uses, with per-extension progress, so the first time you touch a Postgres source or an Iceberg table there is no surprise network hop mid-pipeline:

httpfs (S3 / GCS / HTTP), azure (Azure Blob native), sqlite, postgres, mysql, excel, iceberg, delta, ducklake, vss, fts.

spatial is lazy-loaded (~50 MB GDAL bundle) - it installs on first use of a geospatial source/sink to keep the initial download small.

---

How to use Duckle

A wider tour of the workflow.

---

Recipes and examples

Ready-to-adapt patterns. Each one is a few nodes you wire on the canvas (or ask Duckie to sketch).

CSV cleanup

"Read orders.csv, drop nulls, deduplicate by order_id, write to orders_clean.parquet"

src.csv -> qa.not_null -> qa.uniqueness -> snk.parquet

Set qa.not_null to the columns that must be present; set qa.uniqueness to order_id. Rejected rows go to a snk.csv on the reject port for inspection.

Postgres -> Snowflake nightly load

"Read all rows from Postgres events, upsert into Snowflake table analytics.events on event_id"

src.postgres -> snk.snowflake (mode=upsert, conflict=event_id)

Attach a ctl.schedule with cron 0 2 * * * to run nightly at 02:00.

S3 -> partitioned Parquet

"Read all .json.gz files in s3://logs/2026/*/*.json.gz, parse, write Hive-partitioned by event_date"

src.s3 (glob, autodetect json.gz)
  -> xf.derive (event_date = CAST(ts AS DATE))
  -> snk.parquet (path=out/, partitionBy=event_date, mode=overwrite_or_ignore)

RAG ingestion

"Chunk our docs, embed with OpenAI, dedupe near-identicals, store in pgvector"

src.s3 (markdown files)
  -> xf.ai.chunk (chunkSize=1500, overlap=150)
  -> xf.ai.pii (redact)
  -> xf.ai.embed (model=text-embedding-3-small, baseUrl=https://api.openai.com)
  -> xf.ai.dedupe (threshold=0.95)
  -> snk.pgvector (table=docs)

Slack channel digest

"Pull yesterday's Slack messages from #support, classify by sentiment, email a summary"

src.slack (channels.history with oldest=yesterday)
  -> xf.ai.classify (categories=positive,negative,neutral)
  -> xf.aggregate (group by sentiment, count)
  -> snk.email (to=oncall@..., subject=Daily Support Digest)

Webhook -> S3 archive

"Receive 100 webhooks, archive each one as JSON in S3"

src.webhook (port=8080, maxRequests=100, timeoutMs=300000)
  -> snk.s3 (path=s3://archive/events/, format=jsonl, partitionBy=event_date)

Git commit-log analytics

"Build a dashboard of who's been committing what in the last 30 days"

src.git (mode=log, maxRows=10000)
  -> xf.filter (date > current_date - INTERVAL '30 days')
  -> xf.aggregate (group by author_email, count)
  -> snk.csv (path=author-stats.csv)

More examples live in samples/ - drop the pipeline files into a workspace and open them.

---

Git integration (GitHub + GitLab)

Push, pull, branch, and watch CI from inside Duckle. No terminal required.

Click the Git icon in the topbar to open the workspace Git panel. Built-in integration with GitHub and GitLab, on the system git CLI (no FFI, no embedded git library):

Workflow. Workspaces are plain folders (see Workspace and Git flow) - any standard Git workflow works:

Create / clone -> open in Duckle -> edit pipelines -> commit + push -> 
PR / MR -> CI runs your pipeline tests -> merge -> pull

You can do the entire push / pull / merge loop without leaving Duckle. Heavy operations (interactive rebase, conflict resolution, log archaeology) still live in your terminal or external Git tool - the panel is designed for the everyday flow, not as a full Git replacement.

Provider detection. The remote URL host determines which CI API the badge polls:

The badge uses the same PAT you saved for pushes - no separate auth step.

---

Workspace and Git flow

A workspace is a folder you pick on first launch. Everything you build lives there as plain text:

my-workspace/
  pipelines/
    orders_etl.pipeline.json     # the node graph
    nightly_load.pipeline.json
  connections/
    prod-postgres.connection.json # saved DB credentials (encrypted)
    snowflake-analytics.connection.json
  contexts/
    dev.context.json              # variables for dev environment
    prod.context.json
  routines/
    cleanse-addresses.sql         # reusable SQL snippets
  documents/
    runbook.md                    # plain-Markdown docs
  schedules.json                  # all scheduled runs in this workspace
  run-history/
    orders_etl/                   # one folder per pipeline
      2026-05-25T14-30-00.json    # one file per run

Git-friendly by design. Every file is human-readable JSON or Markdown. Standard workflows work:

git init my-workspace && cd my-workspace
git add . && git commit -m "Initial pipelines"

# Pull a teammate's update
git pull --rebase

# Push your changes
git push

# Branch for a risky migration
git checkout -b feature/upsert-mode
# ...edit pipelines in Duckle...
git diff       # readable JSON diffs
git push -u origin feature/upsert-mode
# open PR / MR

Sensitive values in connections get encrypted with a workspace-local key (workspace/.duckle/keys/). Don't commit that file - add **/.duckle/keys/ to .gitignore. The connection JSON files themselves only hold the ciphertext, which is safe.

---

MCP server (connect Claude or any LLM to Duckle)

Duckle ships its own Model Context Protocol server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor, or any other LLM agent) can drive Duckle directly: browse the full component catalog and per-component property schemas, generate a pipeline straight into a working directory you choose, validate it (compile without running), run it headlessly, read existing pipelines and their run logs, build a standalone artifact, and manage saved connections.

Connect with nothing installed

If you have uv, one line connects any MCP client. Nothing is installed, no engine to configure: uv fetches the package and the DuckDB engine into a throwaway environment and the server finds it there.

claude mcp add duckle -- uvx duckle mcp

For Claude Desktop, Cursor, or any other client, the same thing as config:

{ "mcpServers": { "duckle": { "command": "uvx", "args": ["duckle", "mcp"] } } }

uvx duckle mcp works because the package and the command are both named duckle, so there is no --from to remember. If you would rather install it, pip install duckle puts duckle on PATH and the same duckle mcp command applies.

Then ask the agent something like "use duckle to list the available components". It can discover a real connector rather than guess one, compile-check a pipeline with validate_pipeline before anything executes, run it, and hand back column-level lineage. What it produces is the same JSON the canvas opens, so you can see what it built.

Connect in one click

The MCP server is also bundled inside the app - there is nothing extra to install. In the designer, click Connect to Claude in the top bar to open the connector popup, then pick your client:

• Connect to Claude Code - registers the duckle server for you (runs claude mcp add under the hood).
• Add to Claude Desktop / Add to Cursor - writes the duckle entry into that client's config, with the resolved engine paths filled in (both the Microsoft Store / MSIX and standalone Claude Desktop layouts are handled).
• Or copy the command / config for any other MCP client.

Restart the AI client, then try "Use duckle to list the available components" to confirm the connection.

Manual / headless

For a build-from-source or server setup, point any client at the duckle-mcp binary directly. It speaks JSON-RPC over stdio and reuses the DuckDB engine in-process (no GUI, no Node runtime).

cargo build -p duckle-mcp --release      # target/release/duckle-mcp
claude mcp add duckle -- /path/to/duckle-mcp

For Claude Desktop and other clients, add it to mcpServers:

{
  "mcpServers": {
    "duckle": {
      "command": "/path/to/duckle-mcp",
      "env": {
        "DUCKLE_DUCKDB_BIN": "/path/to/duckdb",
        "DUCKLE_RUNNER_BIN": "/path/to/duckle-runner"
      }
    }
  }
}

Tools: list_components, get_component_schema, create_pipeline, validate_pipeline, run_pipeline, list_pipelines, read_pipeline, read_run_logs, build_pipeline, list_connections, create_connection, backfill_list, backfill_set, backfill_clear. run_pipeline / build_pipeline need a DuckDB binary (DUCKLE_DUCKDB_BIN); build_pipeline also needs duckle-runner (DUCKLE_RUNNER_BIN). Full guide: docs/current/mcp.md.

---

Connection management

Saved connections become DuckDB secrets at runtime so credentials never leak into the pipeline JSON.

Connections live in workspace/connections/ as JSON. The token/password field is encrypted with the workspace key; the rest is plain text.

To use a connection in a pipeline, the Properties panel of any compatible source/sink shows a Connection dropdown - pick one and the fields auto-fill. The list is filtered to connections of a matching kind, so a REST connection is not offered on a JDBC node.

A REST connection is the exception to auto-fill, because it exists to be shared by many nodes that each send a different request: put the vendor's headers and token on the connection once, and rotating a key is a single edit. Headers are merged per key at run time, and the node wins on a key it sets itself; the node's own url and request body are never overwritten. A node with no URL of its own inherits the connection's.

The Copy SQL / Export SQL output is display-only and never executed. Secret values (passwords, tokens, keys, connection strings) are replaced with named placeholders such as ${DUCKLE_PASSWORD}, so the exported script stays valid and is safe to share - substitute the real value at run time. To emit the real credentials instead (so the script runs unchanged), set the environment variable DUCKLE_EXPORT_INCLUDE_SECRETS=1; the output then contains live secrets and should be handled accordingly.

---

Context variables

Bind any field to a context variable that resolves at run time. Useful for dev vs prod, per-environment paths, secrets injected from CI, etc.

In a context file (workspace/contexts/prod.context.json):

{
  "name": "prod",
  "vars": {
    "DB_HOST": "db.internal.acme.com",
    "S3_BUCKET": "acme-prod-data",
    "BATCH_SIZE": "10000"
  }
}

In the Properties panel of any node, switch a field from Manual to Context and pick DB_HOST. Or inline-reference one with ${DB_HOST} in a string field.

Pick the active context from the topbar's Context dropdown. Switch contexts and re-run without editing the pipeline.

---

Build from source

Prerequisites

• Rust (stable)
• Node.js 18+ and npm
• cargo-tauri CLI: cargo install tauri-cli --version "^2"
• Platform webview dependencies per the Tauri prerequisites. WebView2 is preinstalled on Windows 10 and 11.

Clone and install

git clone https://github.com/slothflowlabs/duckle
cd duckle
npm --prefix frontend install

Run in development (hot-reloading frontend plus the native shell):

cargo tauri dev

Build a release binary:

# The --features custom-protocol flag is required: without it, tauri-codegen
# embeds the dev URL instead of the bundled frontend.
cargo build --release --manifest-path apps/desktop/Cargo.toml --features custom-protocol

Outputs land in target/release/duckle (or duckle.exe). The engine is not statically linked: DuckDB downloads at first launch, which is why the build is fast and the binary is tiny.

Run the tests:

cargo test                                                          # workspace unit + plan tests
DUCKLE_DUCKDB_BIN=/path/to/duckdb cargo test -p duckle-duckdb-engine # full integration suite

---

Architecture

duckle/
  apps/desktop/         Tauri 2 shell: Tauri commands, engine installer, llama runtime, window
  frontend/             React 19 + Vite + TypeScript: the designer UI + chat panel
  crates/
    duckdb-engine/      Compiles the node graph to SQL and drives the DuckDB CLI
    slothdb-engine/     SlothDB adapter
    scheduler/          Cron / interval / file-watch triggers
    metadata/           Schema and type model
    plugin-sdk/         Connector / inspector traits
    connectors/         Source and sink connectors
    runtime, workflow-engine, transform-engine, stream-engine, execution-core

• The frontend (React with @xyflow/react) is the visual designer; it talks to the Rust core over Tauri commands.
• duckdb-engine topologically sorts the graph, lowers each node into SQL, and executes by shelling out to the downloaded DuckDB CLI. Non-sink nodes materialize as tables so later stages can reference them; sinks become COPY ... TO statements; cancel kills the process. No statically linked database, so the binary stays small.
• Duckie is a llama-server subprocess on 127.0.0.1 exposing an OpenAI-compatible chat-completions API. The chat panel streams from it via SSE. The model is sandboxed: no fs, no net, no tools - it can only emit text.
• Everything persists to the workspace folder you choose, as plain JSON and Markdown files.

---

Configuration

A few knobs you can set without touching code.

---

Performance tips

A few patterns that consistently produce sub-second runs at small / medium data scale, and tractable runs at warehouse scale.

---

FAQ

Is Duckle free? What's the license?

Yes, free + open source. Dual-licensed MIT OR Apache-2.0. You can use it commercially, fork it, sell what you build with it. No usage limits, no telemetry.

Is Duckle an open-source alternative to Fivetran or Airbyte?

It covers similar ground - moving data across 190 sources and destinations - but locally, with nothing to host and no per-row, per-connector, or per-seat billing. Pipelines are built visually or from plain English and compile to readable DuckDB SQL that runs wherever you deploy it: a laptop, a server, CI or a container. The trade-off is scope: Duckle does not split one query across a cluster, so for warehouse-scale replication you push the work down into the source system or point the output at the system that scales.

Can I run ETL pipelines without the cloud or a data warehouse?

Yes. Duckle executes on the embedded DuckDB engine, so there is no vendor warehouse to buy, no vendor platform to sign up to, and no account. You run it where you choose: a server or VM you own, a container in your own AWS, Azure or GCP account, or a workstation. It needs no outbound network of its own, which suits air-gapped, on-premise and compliance-sensitive work. Pipelines still read from and write to cloud systems whenever you point them at one.

How is Duckle different from Airbyte, dbt, or Talend?

Airbyte focuses on hosted extract-and-load connectors; dbt focuses on SQL transformation; Talend is a heavyweight GUI suite (its free Open Studio edition was discontinued in early 2026). Duckle is a single open engine that does extract, transform, and load together - write it in Python, wire it from connectors, or draw it on a canvas - compiles to DuckDB SQL, and can also run dbt on DuckDB inside the same tool. One format, one engine, running on your own infrastructure rather than a vendor's, with no per-row billing.

Does Duckle send my data anywhere?

No. Duckle makes no outbound calls of its own from wherever you run it, laptop or server. The engines (DuckDB, llama.cpp) are downloaded from official upstream releases on first launch and then run in place. The only network calls Duckle makes on your behalf are the ones your pipelines explicitly do (e.g. a src.s3 reading from your S3 bucket, or xf.ai.embed if you configure it to hit OpenAI).

Duckie needs no network once its model is downloaded - and if you would rather it did not run in-process at all, point it at your own OpenAI-compatible endpoint.

How big are pipelines this works well on?

Bigger than people assume, because the ceiling is the instance you provision rather than the laptop you develop on. The engine is parallel and uses every core available, so the same pipeline that you debug against a sample on a laptop runs against the full set on a large server without changing. For reference, 96M rows come out of live Postgres to Parquet in 39.9s.

Past whatever instance you are willing to pay for, you have two routes that do not involve rewriting anything: turn on pushdown so the query executes inside the source database, or point the output at a warehouse or lakehouse that scales horizontally. What Duckle will not do is spread a single query across a cluster.

Do I need DuckDB installed first?

No - Duckle downloads it for you on first launch. The download is ~30 MB and includes the most-used extensions (httpfs, postgres, mysql, iceberg, delta, vss, fts, etc.) so the first time you touch a Postgres source there's no mid-pipeline network pause.

How big is the binary, exactly?

73 to 110 MB, depending on platform. As of v0.7.0: macOS 73 (x64) to 88 (arm64), Linux 74 (arm64) to 100 (x64), Windows 98 (arm64) to 110 (x64). It embeds the headless runner and the MCP server, and the headless runner on its own is 27 MB. The engines aren't statically linked - DuckDB (~50 MB with extensions) and the Duckie LLM (~1.1 GB for the Qwen GGUF) both download on first launch with a guided installer into your app-data folder, so they update independently of the app.

Can I use OpenAI / Cohere / Voyage instead of the local Duckie?

Yes. The AI transforms (xf.ai.embed, xf.ai.llm, xf.ai.classify) accept a baseUrl prop. Point it at any OpenAI-compatible /v1/... endpoint and an apiKey and Duckle uses that instead. The local Duckie chat panel is hardwired to localhost; the pipeline AI transforms are configurable.

Where does my pipeline data live?

In the workspace folder you pick on first launch (see Workspace and Git flow). Pipelines are plain JSON files you can commit to Git, diff, branch, and review.

Can multiple people collaborate on the same workspace?

Via Git, yes - check the workspace into a repo and use standard branch/PR flows, and deploy the result to a shared server where the console has roles and an audit log. What there is not is a real-time multiplayer canvas: two people editing the same pipeline at the same moment is a merge, not a live session.

Can I run pipelines headlessly / from CI?

Yes. Build Pipeline (right-click a pipeline) produces a single self-contained executable that runs anywhere with nothing installed - drop it on a server or CI runner and execute it, or schedule it with cron / systemd / Task Scheduler. The embedded duckle-runner can also run a workspace pipeline JSON directly (duckle-runner --pipeline pipeline.json). See Server deployment. You can also import the engine crate (duckle-duckdb-engine) into your own Rust binary.

Is the Duckie AI assistant any good?

For 90% of common pipelines (read source -> simple transforms -> sink), yes - the Qwen 2.5 Coder model is tuned for structured-JSON generation. For long, complex pipelines you'll likely want to iterate: describe the first half, click insert, then ask for the next half. You can also swap the model: point xf.ai.llm's baseUrl at GPT-4 or Claude for more capable pipeline drafting.

Does the Duckie panel need internet after install?

No. Once llama-server and the Qwen GGUF are downloaded into your app-data directory, Duckie needs no network at all. Nor does it have to run in-process: point it at your own OpenAI-compatible endpoint and it uses that instead. Tested by killing wifi and asking it for a pipeline - works fine.

Why DuckDB and not Polars / Apache Spark / X?

DuckDB's SQL surface is wide enough to express most ETL work, it's vectorized and fast on a laptop, it has first-class Iceberg/Delta/Parquet readers, and its extension model lets us add vector + full-text + Postgres ATTACH without code changes. Polars is great but doesn't ship the cloud/format/extension breadth we need; Spark is a great cluster but overkill for the local-first niche we're in.

How do I contribute a new connector?

See the Contributing section and crates/duckdb-engine/src/plan.rs (planner branch) + crates/duckdb-engine/src/lib.rs (executor). The shortest path: copy an existing connector with similar shape (e.g. src.rabbit for a streaming source, src.dynamodb for an HTTP+auth API), adapt, add a test, flip the palette tile.

---

Troubleshooting

If you see something not listed, please open an issue with steps to reproduce + the relevant log line.

---

CI / CD

Duckle's CI pipeline runs on both GitHub and GitLab - the project mirrors to both. Push / pull-request / merge-request / tag events all trigger builds.

What each pipeline does:

1. Frontend - npm ci + npm run build (type-check + bundle)
2. Rust test matrix - cargo test --workspace on Linux + macOS + Windows
3. Live-service integration tests - PostgreSQL + MySQL + MinIO services spun up via Docker, real connector code runs against them
4. Desktop release-build smoke check - cargo build --release --features custom-protocol then grep the binary for the embedded frontend JS chunk (catches the v0.0.7-class "binary loads devUrl" bug at PR time)
5. Format + clippy - informational (does not block merge)
6. On tag: build the Duckle binary on all three OSes, upload as release assets

See .github/workflows/ and .gitlab-ci.yml for the exact steps. The two pipelines are kept feature-equivalent so contributors can fork to either platform.

Releasing a new version

Nothing regenerates this README, the hero / flow SVGs, or the download links automatically - they are hand-maintained, so they drift unless each release updates them. Treat the README as a release artifact: walk this checklist every time before tagging.

# 0. Update the README in the SAME commit as the version bump:
#    - bump every vX.Y.Z reference (the Download / Install link, badges)
#    - refresh capability tables for any new sources/transforms/sinks
#    - add/replace screenshots in docs/assets for shipped features
#    - re-check the hero/flow SVG wording if positioning changed
# 1. Bump version in apps/desktop/tauri.conf.json
# 2. Commit (README + version together)
git commit -am "Release: bump to vX.Y.Z"
# 3. Tag + push
git tag vX.Y.Z
git push origin main vX.Y.Z
# Both GitHub Actions and GitLab CI pick up the tag and build the
# release artifacts automatically. Once green, the draft release on
# GitHub gets the binaries uploaded; un-draft + mark Latest with:
gh release edit vX.Y.Z --draft=false --latest

---

What's new in v0.7.0

A server somebody can set up in a browser, an ordered plan of pipelines, a catalog of the whole workspace, and a run that stopped reading the source three times to answer one question.

• Set a server up without a terminal. The console used to assume whoever deployed it would also configure it over SSH. It now answers a first visit with a setup page: pick where the server runs, claim it, and the administrator token is shown once. Claiming is the whole setup. A liveness probe at /healthz means an orchestrator can tell a starting server from a wedged one, and a schedule that stops working now says so instead of failing quietly.
• Sign-in, roles and an audit log. Accounts and machine credentials live in SQLite rather than a file two processes could erase for each other. People are managed from a People tab instead of a deployment task, machines get their own API keys, and sessions survive a restart, expire, and travel marked. Roles are enforced in one place per surface, so a route cannot be reached by forgetting to check it at the call site. The audit log records who did what, including refusals, and can be read back from the console.
• Plans: several pipelines, in an order somebody chose. A plan is a list of pipelines that run in sequence. Plans are authored in the desktop app and in the console, share one plans.json, and can be scheduled like a single pipeline. A plan whose pipelines failed no longer reports that it worked.
• A Data Catalog across pipelines, not inside one. The workspace graph now spans pipelines: which asset feeds which, what is orphaned, who owns it, and how fresh it is, because every run records what it touched. Columns, descriptions, tags and a glossary are editable in a Data Catalog screen, exposed to the console and to agents through MCP, and a saved graph can tell you when the pipelines have moved on.
• Queued work, and a worker that claims it. A ForEach can dispatch its items as a batch rather than running them here. A worker claims queued items, the console shows what is queued and lets you retry what failed, and the queue says whether items can safely run at once. Each sub-pipeline runs under its own name and keeps its own watermark.
• Runs stopped scanning the source three times. Every node's row count was a separate SELECT COUNT(*), and since nodes are views, each one re-ran the whole chain. A source to filter to sink pipeline read a 96M-row Postgres table three times to do one pass of work. Each relation is now counted once, and a sink takes its count from the Parquet footer of the file it just wrote, which is a metadata read: 0.06s against 16.7s for the equivalent count over the source. Measured on that pipeline, baseline against this release, interleaved on one machine: 56.3s to 18.8s, and 288,159,946 tuples scanned down to 96,011,803. That puts it level with a hand-written DuckDB COPY doing the same work, at 1.02x. A remote XML stream over SFTP was reading 8 KiB per round trip and now reads 256 KiB: 75 MB and 700,000 rows went from 17.0s to 10.0s.
• Security fixes, two of them serious. The streaming run route accepted work without authentication, and the in-app updater pointed at a GitHub organisation nobody had registered, so whoever claimed the name could have served the next update. Both are closed. Beyond those: connection secrets are bound to the field and connection they belong to, so a ciphertext cannot be moved between fields; bundle keys are derived with Argon2id and a per-bundle salt instead of an unsalted hash; downloaded engines and models are verified against a pinned checksum; run parameters can no longer redefine builtins or inject shell syntax; sidecars stage in a private directory instead of shared temp; decrypted connection secrets stay out of browser storage; a cached git token is re-encrypted and never written world-readable; and a deploy refuses to send a pipeline carrying a credential in plain text.
• Saved connections are a reference, not a copy. Picking a saved connection used to copy its values onto the node, so the credential was duplicated into the pipeline file and a later edit to the connection did not reach it. The node now stores only the reference and resolves it at run time. The editor shows what the connection will supply rather than the manifest's default, so a node pointing at a connection on a non-standard port stops displaying the standard one, and a secret says only that it is covered.
• Home is a launcher, and the tour goes first. The app opens on three tiles rather than dropping you into a canvas, modules are one level in, and a first run walks every capability once instead of showing the tour and Home at the same time. Settings can ask again.
• Deploying is documented, including the uncomfortable parts. A deployment guide for AWS, Azure and Google Cloud, the client and server architecture, promoting from CI, driving Duckle from another orchestrator, and a walkthrough of the whole server flow. The docs now say plainly that a failed run still answers 200, before somebody trusts it.

---

What's new in v0.6.1

Talend jobs import straight into the canvas, and credentials are masked more carefully in exported SQL.

• Import a Talend job from the editor. A Talend button sits in the project sidebar next to New Pipeline and New Folder, and the same action is in the editor's ⋯ menu as Import Talend job.... Either one picks a .item job, translates it, and opens it as a new pipeline tab, laid out on the canvas at the coordinates the job was drawn with. Measured on a real 44-job corpus: all 44 parse and 211 of 216 nodes map, the only refusal being a site-specific custom component. Nothing is written to your workspace until you save, so a job that translates badly costs a closed tab.
• The import report says what still needs a person. A node count on its own would suggest a working pipeline, so the report leads with how many components actually translated, then lists everything unresolved. Encrypted Studio passwords arrive as ${ENV:...} placeholders instead of guesses. Connections stored outside the job file are named, so you can fill them in or point the node at a saved connection. tMap outputs computed by Java are listed column by column with the expression to rewrite as SQL. A component with no Duckle equivalent is imported as a labelled placeholder, so the shape of the job survives rather than quietly losing a step.
• Convert a whole repository at once, from the terminal. Importing through a file dialog is the right shape for trying Duckle and the wrong shape for leaving another tool, because nobody has one job - they have several hundred in a checkout. duckle-runner import <dir> walks the tree, converts every job it finds, and mirrors the folder layout under --out so two jobs that share a name cannot overwrite each other. Measured on a real 125-file corpus: 42 files hold a job and 83 do not (routines, contexts and SQL templates share the extension), all 42 convert with none failing, and exactly one component across the whole corpus has no equivalent - a site-specific custom one. Everything else still to resolve is credentials that were never in the job files to begin with: 119 encrypted passwords and 75 connections defined outside the job. The closing tally lists unmapped components by how often they appear, which is both the answer to "is this migration viable" and the shortest path to finishing it. --json for a script, --strict to fail a CI job.
• A file that was never a job is skipped, not counted. The extension is shared by routines, contexts and SQL pattern templates. Parsing those yields an empty pipeline, and counting an empty pipeline as a converted job inflates the only number anyone reads - on the corpus above it turned 42 real conversions into a reported 82. They are now reported separately and no empty pipeline is written. The same rule applies in the other direction: a routine is Java source whose javadoc breaks any XML reader, so it is skipped rather than reported as a failed job, while a file that declares itself a job and then will not parse is still a failure.
• Credentials are masked on token boundaries in exported SQL. Redaction replaced a credential value wherever it appeared, so a password that was also a substring of an ordinary identifier corrupted the statement around it: a password of prod rewrote production_report.parquet as ${DUCKLE_PASSWORD}uction_report.parquet. The secret itself was always protected; the damage was to everything else, and it mattered most when reading the Plan or SQL view to debug. Matching is now delimiter-aware, so LOAD postgres is left intact while a one-character password is still masked in password=p'. Deliberately no minimum length: a short password is still a password.

---

What's new in v0.6.0

A multimodal AI data store, an importer for legacy visual ETL jobs, a chat model you choose, and two geometry transforms that finally have the second input they always needed.

• Pixeltable, read and write (#223). src.pixeltable reads a table, optionally filtered by a Pixeltable expression, a column subset and a limit; snk.pixeltable inserts into an existing table or creates one from the incoming rows. Versioned reads work by passing myapp.media:3. The exchange runs over Parquet on both legs - Pixeltable exports, Duckle ingests with read_parquet, and on the way back Duckle writes Parquet that Table.insert takes directly - so no rows are serialised one at a time. Pixeltable is a Python library, so the desktop app provisions a private Python for it with uv on first use; nothing is installed into your own environment.
• Clip and Erase can now be wired up (#217, #218). Both shipped in v0.5.9 as two-layer overlays, and the engine required the second layer, but the palette declared only one input - so the node could be placed and configured and never run. Both now offer a second input labelled clip layer / erase layer, like Spatial Join. Behaviour is unchanged: the second layer is still dissolved with ST_Union_Agg before the operation, attributes of the input layer are preserved, and features left with nothing are dropped. Thanks to @OmarMustaafa for reporting it twice with screenshots. A test now pins this contract for every component whose builder needs a second input, checked by removing a port and confirming it fails.
• Choose the assistant's model, from 14 (#223 adjacent). The setup step installed one hardcoded 1.5B model, which is right on a laptop and wrong on a workstation with a GPU. The catalogue now spans 469 MB to 9.9 GB - Qwen2.5 Coder 0.5B through 14B, Qwen3, Llama 3.2, Phi-3.5 Mini, Mistral 7B, Gemma 2 9B and DeepSeek Coder V2 Lite - each with its real download size and an honest note on what it needs. Every entry was checked to resolve before being offered, so the picker cannot hand you a file that 404s halfway through a multi-gigabyte download.
• Import jobs from legacy visual ETL tools. Reads the XML those jobs are stored as and produces a Duckle pipeline. Measured on a real 44-job corpus: all 44 parse and 211 of 216 nodes map, the only refusal being a site-specific custom component. Encrypted passwords become ${ENV:...} placeholders rather than guesses, connections that live outside the job file are reported rather than silently half-imported, and anything with no equivalent is imported as a labelled placeholder so the shape of the job survives instead of quietly losing a step.
• CI for your pipeline repo. duckle-runner is now published as a release asset, and there are ready workflows for GitHub Actions and GitLab CI under docs/ci/. They gate every push on duckle-runner validate, which compiles pipelines to SQL without opening a source, writing a sink, or needing credentials or a network. This is the check that catches a column renamed in one commit and still referenced by another - they merge cleanly, and nothing else notices.
• Node ids no longer collide. Adding a folder or duplicating an item minted an id from the clock alone, so two of the same kind created in the same millisecond could share one. Both now use the same timestamp-plus-random scheme as everything else, which is what makes pipeline JSON safe to merge across branches.

Full notes: see the v0.6.0 release.

---

What's new in v0.5.10

Power mode, context layering, and an Oracle extract that is now faster than python-oracledb with pyarrow on the same table.

• Power mode (Settings -> Power mode). Two throughput settings per workspace. Pipelines at once caps how many run together; the placeholder shows the machine's core count. Spill folder points DuckDB's spill files at a bigger or faster disk. Only the lever with a measurement behind it is offered: independent pipelines scaled about 3.8x across 8 concurrent processes on a 20-core box, while splitting a single pipeline across processes measured slower (72ms to 123ms at 8-way), so there is deliberately no option for it. Each concurrent run gets its own memory limit and its own DuckDB process, so N at once needs roughly N times the memory, and the panel says so.

• Scheduled runs have a ceiling. Every schedule that came due in the same tick fired at once, so ten due at midnight meant ten pipelines each sized for the whole machine. They are now bounded, by power mode where it is set and by a sane default otherwise. The headless duckle serve honours the same setting, so desktop and server agree.

• Contexts can be layered (#204). A context can declare a Layer; higher layers override lower ones. A shared base plus a per-environment override is now expressible directly: give the base layer 0 and the environment a higher number, and the override applies quietly. Previously all contexts merged flat in repo order, so every intended override looked like a collision and had to be resolved by hand. Only two contexts on the same layer defining the same name are still reported, because nothing there says which should win. Workspaces that set no layers merge exactly as before.

• Oracle extracts beat python-oracledb (#221). Three changes, each measured on a 1,466,723-row x 236-column table with the same query and SNAPPY on both sides:

  • Unconstrained NUMBER columns are now measured before the write instead of typed after it. Those columns have no declared width, so they used to travel as text and be typed by a pass over the finished Parquet - which is exactly the pass a direct write skips, meaning one such column forced a whole second pass over every column. The ambiguous columns are now read on their own first (about 2s for 4 of 236), their real widths pin the schema, and the file is written once. Both reads share one snapshot via a read-only transaction, so they cannot disagree.
  • The driver no longer rebuilds an owned row per fetch. ResultSet<Row> reconstructs every value in the row; on this table that was 346 million reconstructions, measured at 11.5s of a 42.7s fetch against a 31.2s floor.
  • Scaled NUMBER values no longer allocate a string per cell while being rescaled - 88 million allocations per run on this shape.

  Together: 100.7s to 65.0s in the shape reported on #221, against 68.6s for python-oracledb with pyarrow doing the same job on the same machine. With column types already pinned it is about 56.7s. Output was verified against pyarrow's: identical row counts, and equal sums, hashes, ranges and null counts across every column type. Worth noting that python-oracledb maps an unconstrained NUMBER to DOUBLE, which cannot hold the 24 significant digits one test column carries; Duckle types it exactly, so the comparison is not quite like for like and not in our favour.

• A direct Parquet write no longer produces string columns (#221). With Write directly from the source enabled on a table containing any bare NUMBER, those columns were written as text while the run reported success. The source now declines the shortcut when a column cannot be typed before the write, and says so in the run log. Anyone who enabled that toggle on v0.5.9 against such a table should re-check the output.

• Concurrent runs no longer fight over spill files. DuckDB's default spill location is already per-run, but setting a shared spill folder made every run share one - which reads as a flaky run rather than a bug: across three trials of four concurrent spilling queries, a shared folder lost 3 of 12 runs to a segfault or a delete failure, private folders lost 0 of 12. Each run now spills into its own subfolder.

Full notes: see the v0.5.10 release.

---

Roadmap

A complete planned-component breakdown lives in docs/roadmap.md. Highlights:

• Multi-shard Kinesis and Pulsar streaming (Pulsar blocked on protoc at build time)
• Apache ORC read / write (blocked on the Arrow version conflict between orc-rust and our workspace pin)
• SFTP source (shipped - russh + russh-sftp on the ring backend, password / key auth, host-fingerprint pin)
• OAuth-heavy SaaS (Google Sheets, Excel Online, full Salesforce OAuth, Gmail / O365 IMAP)
• Embedded Python / Rust code stages (current code.* family: SQL, Shell, JavaScript, WebAssembly all ship)
• Hosted documentation site
• Plugin marketplace via the connector SDK
• In-process Native engine - a Rust streaming / incremental executor as an alternative to shelling out to the DuckDB CLI

---

Contributing

Contributions, issues, and ideas are welcome. Duckle is young and there is a lot of green field. Open an issue to discuss a change before a large PR, match the existing code style, and keep changes focused. Run cargo test and npm --prefix frontend run build before submitting. See CONTRIBUTING.md.

---

Contributors

Thanks goes to these wonderful people who contribute to Duckle (emoji key):

This project follows the all-contributors specification. Contributions of any kind - code, docs, design, bug reports, ideas - are welcome and recognized here. Comment on any issue or PR with @all-contributors please add @name for code, doc and the bot opens a PR adding them.

---

License

Licensed under either of MIT or Apache-2.0 at your option.

---