---

[!TIP] TL;DR — 100+ self-contained data-engineering projects. Real algorithms, 2 500+ passing tests, mypy --strict, multi-stage Dockerfiles, GitHub Actions matrix on Python 3.10/3.11/3.12, MIT-licensed. Zero or minimal runtime dependencies. Browse the directory ↓.

A curated monorepo of data-engineering work — pipelines, storage engines, query engines, table formats, streaming, governance, and distributed-systems primitives. Each subdirectory is an independent project with its own README, tests, Docker image, and CI workflow.

✨ Why this repo

• 🧱 Production-grade, not toy examples. Every project ships with strict type-checking, ruff lint, hand-written tests, Docker images, and GHA CI.
• 🧬 From-scratch implementations of the algorithms that power Kafka / Iceberg / Delta / Hudi / Parquet / CDC / Postgres WAL — read the code, learn how they actually work.
• 🐍 Zero or minimal runtime dependencies — most projects use only the Python stdlib. The wire formats are real; the code is auditable.
• ⚡ One-command local runs — cd <project> && pip install -e ".[dev]" && pytest works for every project. No external Kafka / Postgres / cluster required.
• 📚 Pedagogical clarity — every module has a top-of-file docstring explaining the algorithm, the trade-off, and the failure modes.

🌟 Featured Projects

If you only look at five, look at these:

🚀 Quick start

# Pick a project, install, run its tests, see the demo:
git clone https://github.com/sophie-nguyenthuthuy/data-engineering.git
cd data-engineering/log-based-cdc-from-scratch
pip install -e ".[dev]"
pytest -q
lcdcctl info

That same recipe works for every project in the monorepo. Each one has a Makefile with install, dev, test, lint, type, docker targets.

📊 Stats

• 159 self-contained projects across ingestion, storage, processing, governance, and platform — 105 curated in the directory below, the rest in the active build series (CHANGELOG).
• 79 production-grade builds + 21 research MVPs in the curated directory.
• 2 500+ tests (each project ships its own suite — see per-project README).
• mypy --strict clean on every project.
• Python 3.10 / 3.11 / 3.12 CI matrix per project.

🤝 Contributing

PRs welcome — see CONTRIBUTING.md. Good first issues:

• Add a new edge case to any project's test suite.
• Wire a project up to a real backend (Kafka / Postgres / S3) behind its existing injectable adapter.
• Add benchmarks comparing against pyarrow / fastavro / etc.

🔍 Keywords

_{data engineering • lakehouse • apache iceberg • delta lake • apache hudi • parquet • orc • avro • change data capture • CDC • Postgres WAL • MySQL binlog • pgoutput • Kafka • Kafka Streams • Flink • streaming • event sourcing • exactly-once • watermarks • feature store • differential privacy • DBT • Airflow • Dagster • Prefect • semantic layer • cube • metric store • B-epsilon tree • LSM tree • ART • MVCC • Raft • Paxos • CRDT • vector clocks • HLC • hybrid logical clocks • TLA+ • Jepsen • linearizability • causal consistency • property-based testing • Hypothesis • approximate query processing • coresets • KLL sketch • Bayesian data quality • shuffle DP • k-anonymity • data mesh • data contracts • OpenLineage • schema registry • Iceberg time travel • Delta time travel • predicate pushdown • Z-order • Hilbert curve • Bloom filter • Cuckoo filter • IVM • differential dataflow • timely dataflow • Naiad • Volcano • Cascades query optimizer • cost-based optimizer • cardinality estimation • TPC-H • TPC-DS • MinIO • Trino • Presto • ClickHouse • DuckDB • Polars • Spark • Python 3.12 • mypy strict • ruff • from scratch • production-grade • monorepo}

---

Directory

Built (79):

• Streaming & Real-time (15)
• Batch, ETL & Orchestration (18)
• Data Quality & Governance (11)
• Query & Storage Engines (17)
• Data Warehouse & Lakehouse (8)
• Distributed Systems & Infrastructure (10)

Research MVPs (21):

• Query Processing & Compilation
• Distributed Systems Theory
• Storage Engine Internals
• Streaming & Time
• Data Correctness & Formal Methods
• ML-Data System Co-design
• Platform & Meta-Systems

---

Streaming & Real-time

Batch, ETL & Orchestration

Data Quality & Governance

Query & Storage Engines

Data Warehouse & Lakehouse

Distributed Systems & Infrastructure

---

Research MVPs

Working implementations of 15 research-grade project ideas. Each has:

• Core algorithm in src/ (3–6 modules, 200–1000 LOC of Python)
• Passing test suite (5–9 tests per project, all green)
• Runnable demo in examples/ showing the system in action
• Makefile with install, test, demo targets

Aggregate: 106 tests pass across the 15 MVPs.

Pedagogical (single-process Python) — not production clones of HyPer/Naiad/DuckDB. The core algorithms are real and tested.

Query Processing & Compilation

Distributed Systems Theory

Storage Engine Internals

Streaming & Time

Data Correctness & Formal Methods

ML-Data System Co-design

Platform & Meta-Systems

---

Layout

data-engineering/
├── README.md               ← this file
├── <project>/              ← each project is an independent repo with its own .git
│   ├── README.md
│   └── ...
└── ...

Notes

• All cloned via git clone --depth 1 (no full history). The originals on GitHub still hold full history.
• bitcoinMonitor/ is a local MVP. Stack: CoinGecko poller → SQLite → FastAPI + Chart.js dashboard.
• Research MVPs are working implementations — cd <project>/ && make install && make test && make demo. Pedagogical (Python, single-process) not production (no actual LLVM JIT, RDMA, Kubernetes). The 6 "Built" entries cross-referenced in those sections link to the corresponding fuller implementations.
• Each project's runtime is self-contained — see its README for setup.