Workflow Orchestration and Scheduling

17 repos

Platforms and frameworks for defining, scheduling, and orchestrating complex data pipelines and automated workflows. This cluster covers both Python-based solutions (Airflow, Prefect) and polyglot engines (Java and C# implementations like Kestra and Wexflow) that enable users to build DAG-based task dependencies, monitor execution, and manage data engineering workloads at scale. Repositories include core orchestration engines, official documentation, and community forks that serve teams automating batch processing, ETL, and distributed task scheduling.

Python · 6

Go · 4

Java · 3

Astro · 1

C# · 1

Ruby · 1

workflow ·192,122

orchestration ·176,582

data-engineering ·167,042

automation ·146,619

etl ·146,116

scheduler ·138,928

data-orchestrator ·138,086

workflow-engine ·137,348

python ·135,888

data-science ·133,797

Related clusters

Workflow Engines and Orchestration13 Python Data Science & Analysis38 Cluster 63357733 Cluster 63642411 Cluster 63479936 Infrastructure as Code & DevOps Tooling18

apache/airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Python

46,884

33,718 commits

apache/incubator-airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Python

46,886

33,743 commits

kestra-io/docs

Documentation for Kestra — an event-driven, language-agnostic orchestration and scheduling platform…

Astro

155

6,154 commits

PrefectHQ/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Python

23,859

20,352 commits

aelassas/Wexflow

Workflow Automation Engine

842

978 commits

dagster-io/dagster

An orchestration platform for the development, production, and observation of data assets.

Python

16,168

19,267 commits

kestra-io/kestra

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Java

28,148

11,639 commits

flowable/flowable-engine

A compact and highly efficient workflow and Business Process Management (BPM) platform for…

Java

9,541

10,315 commits

floraison/flor

a workflow engine

Ruby

287

3,490 commits

stjude-rust-labs/sprocket

A bioinformatics workflow engine built on top of the Workflow Description Language (WDL).

Rust

210

1,103 commits

apache/dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance…

Java

14,482

7,823 commits

lllons/NoteTaker

AI SST model that takes all the noise from you classes and then gives it to you with time stamps…

Python

13 commits

mara/data-integration

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Python

2,091

165 commits

ergo-services/ergo

An actor-based Framework with network transparency for creating event-driven architecture in…

4,660

211 commits

ConduitIO/conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

611

1,760 commits

rudderlabs/rudder-server

Privacy and Security focused Segment-alternative, in Golang and React

4,486

6,179 commits

dagucloud/dagu

Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML…

4,024

3,055 commits