Lakehouse Industry Data Models

Forty production-ready industry data models, each shipped in two flavours (ECM + MVM) — auto-generated by the Vibe Data Modeling AI agent found here under model-agent/ that turns a "user vibes" into a Unity-Catalog-ready model with schemas, foreign keys, metric views, ontology tags, and DBML diagrams. The installer can also populate every table with referentially-correct sample data on the way in.

40 industries · 80 models · 23,092 tables · 885,842 attributes · 156,641 foreign keys · 11,661 metric views

Open the interactive gallery

---

At a glance

MVM ≈ 39% of ECM by table count, but retains 44% of the attributes and 59% of the FK relationships — the MVM keeps the join-heavy entities and sheds low-traffic reference tables.

---

How to install a model

Install any model into Unity Catalog — catalog, schemas, tables, foreign keys, governance tags, and metric views — with the model-installer/data-model-installer.ipynb notebook. It is Databricks Serverless compatible (every operation is a plain spark.sql call).

Steps

1. Import the installer. Download model-installer/data-model-installer.ipynb and import it into your Databricks workspace (or clone this repo into a Git folder so the whole model-installer/ folder comes with it).
2. Pick an industry and click Run All. Open the model widget, choose an industry, and run the notebook. Everything else has a sensible default — no other input required.
3. Watch it install live. With session_id left blank, the notebook launches itself as a Databricks job and prints the run URL so you can follow the install in real time. Progress is timestamped in dependency order: catalog → schemas → tables → foreign keys → tags → metric views. Any failed statements are retried automatically at the end.
4. Done. The model lands in the <catalog_name> catalog (defaults to the industry name), ready to query.

Installing with sample data

An empty model is hard to demo, so the installer can fill it. Set generate_samples to Yes and pick sample_rows (5 / 10 / 20 / 50 / 100, default 10). Nothing else changes: samples are the last phase, after tables, foreign keys, tags, and metric views, and they are skipped automatically if the structural install left failures behind.

The generated data is safe to join and safe to demo on:

The structure is read back from information_schema after the install completes, so generation targets the tables, keys, and relationships Unity Catalog actually holds rather than what the model file declared. Views, metric views, and internal schemas (_metrics, _install, information_schema, default) are never populated.

An optional pass asks a Databricks Foundation Model endpoint for realistic value pools for free-text columns (names, descriptions, cities). It is time-boxed per table and never used for keys, so a slow, unavailable, or nonsensical answer costs that table some realism and nothing else.

Verified end to end on live installs, audited directly against information_schema rather than trusted from the generator:

The banking install is the harder case: 227 tables at the maximum row count, with 81% of its relationships crossing a schema boundary.

Widgets

The installer shows nine widgets, in order:

Advanced settings are not shown as widgets and use built-in defaults forwarded to the launched job automatically: 32 threads × 20-statement batches (the measured serverless optimum), metric views on, and source = this repo. session_id is job-injected — leave it blank on an interactive run and the notebook launches the install as a Databricks job that sets it automatically to run the install in-place. Sample generation adds four more: sample_seed (default 20260801), sample_llm (true), sample_llm_endpoints (comma-separated Foundation Model endpoints), and sample_threads (8).

The launched job tags itself (prefix dbx_vibe_agent_installer_) with the industry, size, version, and final install duration. The notebook's first cell documents every widget, phase, and sample-data guarantee in full.

---

How to view a model

The fastest way to explore one of these models visually is the model-viewer app, a Databricks App that renders any model.json as an interactive entity-relationship graph with three navigable views (full model, domain, single product).

Step 1 — Install the viewer app

1. Download the installer notebook from this repo: model-viewer/model_viewer_app_installer.ipynb.
2. Import the notebook into your Databricks workspace and run all cells. The installer provisions a Databricks App and prints the app URL when it finishes.

Step 2 — Load a model

Open the app URL. You have two ways to load any model from this repo:

• Load from repo — paste databricks-industry-solutions/lakehouse-industry-data-models and pick the industry + flavour from the dropdown. Note: GitHub sometimes rate-limits anonymous API calls — if you hit a 429 / "rate limit exceeded" message, fall back to the second option.
• Load from JSON — navigate to the industry folder in this repo (e.g. data-models/retail/v1/mvm/), download model.json, and click Load from JSON in the app to upload it directly.

What you see in the viewer

Full-model overview — every entity in the model arranged on a single canvas, with every foreign-key relationship drawn between them. Domains are colour-coded (each rectangle is one domain) and products sit on the perimeter:

Domain drill-down — click any domain to zoom in. You see the domain's sub-domains as named groups and the products inside each, with the FK web restricted to within-domain links:

Single-product radial view — click any product (table) to centre it. The viewer fans out every other product it relates to via FK, grouped by domain, so you can see at a glance every join path leaving that table:

---

How models are generated

Every model in this repository was generated by the Vibe Modeling Agent, found here under model-agent/. Point it at a plain-English description of a business and it produces a governed, Unity-Catalog-ready model; the 80 models above are its output.

Learn more: Reimagining Data Modeling on the Lakehouse: Introducing Vibe Data Modeling and Jumpstart your Data Modeling with Databricks Industry Data Models.

1. An LLM-powered modelling AI agent receives user vibes (e.g. "create 7 domains covering the arilines, ensure flight is one of them, cover airport operations too").
2. It runs an 8-stage AI agent using a per-stage LLM ensemble + judge:
   1. Tier classification → 2. Domain generation → 3. Sub-domain expansion → 4. Product (table) generation → 5. Attribute (column) generation → 6. FK linking → 7. Semantic dedup + naming → 8. Metric view + ontology synthesis.
3. Each stage is gated by structural validators (cycle detector, bidirectional-FK detector, dangling-FK detector, fidelity-precision gate) before the next stage starts.
4. Output is written to a Unity Catalog volume + workspace folder, then published into this repo.

---

What you get per industry

Each industry root folder ships two flavours of the same business domain:

• <version>/ecm/ — Expanded Coverage Model. Comprehensive, audit-grade model — the agent's source of truth. Covers every entity it can think of for the industry: operations, finance, regulatory, audit trail, reference data.
• <version>/mvm/ — Minimum Viable Model. Production-ready, demo-friendly subset derived from the ECM. Roughly 40% of the ECM's table count, retaining the most-used entities and FK paths. Recommended starting point for new deployments.

Twenty-six industries are on v1; fourteen have a v2 produced by a later agent. The installer resolves the highest version present, so you get the newest model unless you point it at an older folder yourself.

Both flavours are byte-identical in shape — same files, same structure, same Unity-Catalog deployment story. You pick the size that fits your use-case.

---

Repository layout

<industry>/
└── v1/                          # Model generation version (v2/, v3/, … land as siblings)
    ├── readme.md                # Industry-level summary (counts, vibe, generation metadata)
    ├── ecm/
    │   ├── readme.md            # ECM-specific summary + per-domain breakdown
    │   ├── model.json           # Full agent model (single source of truth)
    │   ├── schemas/             # Per-table DDL (CREATE TABLE / CREATE VIEW)
    │   ├── metrics/             # Metric view SQL (one .sql per BI-ready metric view)
    │   ├── ontology/            # Tag taxonomy + classification ontology JSON
    │   ├── docs/                # Per-domain markdown docs (auto-generated)
    │   ├── diagram/             # DBML + auto-rendered ER diagrams
    │   └── vibes/               # next_vibes.txt — auto-generated improvement priorities for the next version
    └── mvm/
        └── (same structure as ecm/)

Each new model generation lands under a new vN/ sibling (v2/, v3/, …), so multiple versions of the same industry stay side-by-side and reviewable.

Sample data is not shipped as files. It is generated at install time from the physical catalog, so the rows match the keys and relationships Unity Catalog actually holds. A few older model folders still carry a samples/ directory from an earlier generation; it is not used by the installer.

Top-level helper:

• data-models/models-info.csv — flat machine-readable manifest of every model, one row per industry. The version column names the shipped version each structural count describes. The cost, duration, and token columns record the original generation run and were not re-measured for the fourteen industries that were later regenerated.

---

Quality gates — every model passes

Every shipped model was validated against the agent's §9 model-level integrity contract. Findings are split out per flavour so you can see the MVMs are entirely structurally clean.

All 40 MVMs ship with zero structural findings — clean across every check above. The 15 ECM silos and 34 ECM cross-domain name overlaps are the only outstanding items, all minor and called out in Known limitations. Cycles, bidirectional pairs, dangling FKs and self-FKs are zero across all 80 models.

---

Headline highlights

Top-5 biggest ECMs by attribute count:

Top-5 biggest MVMs by attribute count:

Most relationship-rich (densest FK graph):

Most BI-ready (most metric views):

Deepest sub-domain hierarchy:

---

Industry index — full catalog

Click an industry name to jump to its folder.

Financial Services & Insurance

Healthcare & Life Sciences

Travel & Logistics

Energy & Resources

Public Sector & Services

Communications, Media & Entertainment

Retail & Consumer

Manufacturing & Industrial

---

Provenance

• Agent versions used: v4.5.5 (1 industry), v4.5.3 (1 industry), v4.4.7 (1 industry), v4.3.3 (2 industries), v4.3.2 (9 industries), v0.7.2 (7 industries), v0.7.1 (19 industries) — single-digit semver per CLAUDE.md §3a. The 14 industries carrying a v2/ were regenerated by the 4.x agent; the rest are still on their original 0.7.x generation.
• Every model.json carries a top-level agent_version field so you can correlate model shape to the producing agent revision.
• Every commit is one industry — search git log --oneline | grep <industry> to find it. Diffs across versions of the same industry stay reviewable.
• data-models/models-info.csv is the flat machine-readable manifest of all per-model metrics.

---

Known limitations

• Metric views install cleanly. An earlier generation confused some column names in a handful of metric views, which caused occasional install errors. Every affected reference has been corrected against the physical schema — column renames, two source-schema corrections, and one nested-aggregate rewrite — with no views dropped. All 80 models now install with zero metric-view errors, validated end-to-end against live catalogs across all 40 ECMs and 40 MVMs.
• Sample data is synthetic and opt-in. The installer generates it only when generate_samples is Yes. Keys and foreign keys are correct by construction and gated before the write, so joins and demos behave, but the values themselves are invented. Never treat them as ground truth for analytics; replace with real ingestion before going to production.
• 18 ECMs carry 34 cross-domain duplicate product names (e.g. party owned by two domains in payments_fintech/v1/ecm, legal_entity in banking/v1/ecm). These are usually legitimate shared lookups, but a future agent version may consolidate each under a single owning domain. All 40 MVMs are clean of this.
• 11 ECMs contain 15 siloed products between them (one or two each, e.g. finance.ledger in education/v1/ecm, inventory.uom in restaurants/v2/ecm, audit.cae_charter in banking/v1/ecm). These are legitimate top-level reference entities the agent chose not to link out from. All 40 MVMs are silo-free.
• Industry coverage is broad, not deep. The ECMs aim for 70-80% of an enterprise's domain shape; the last 20-30% (organisation-specific extensions, third-party integrations) is a follow-up vibe-iteration the agent can take on.

---

License

These models are auto-generated and provided as-is for reference. Industry standards evolve; verify against your organisation's specific business rules and regulatory context before production use.

---

40 industries · 80 models · 23,092 tables · 885,842 attributes · zero FK cycles, zero dangling foreign keys across all 80.

---