README
Awesome Data Contracts
Data contracts is an amazing initiative to bring data producers and data consumers together. This is a curated list of awesome articles, tools, resources that will help you understand concepts and start your data contracts journey.
Articles and Blogs
- Why the Need for Standardizing Data Contracts? The genesis of Bitol or why data contracts are an essential part of modern data engineering.
- Data Contract 101 A quick and not-so-dirty introduction to data contracts, focusing on the Open Data Contract Standard (ODCS).
- Improving data quality with data contracts (GoCardless) What are data contracts and GoCardless implementation of data contracts
- Interfaces and Breaking Stuff by Tristan Handy (CEO, DBT) Need of data contracts as data interfaces and analogy with software APIs
- Implementing data contracts at GoCardless Data contract specification examples from GoCardless systems
- 7 key learnings from implementing data contracts (Bar Moses and Andrew Jones) Sample data contract example and learnings from implementations
- A PM's thoughts on data contracts A product manager’s view on data contracts and challenges they can solve
- Engineer’s guide to data contracts part I - entities covers implementing data contracts for entities
- Engineer’s guide to data contracts part II - application events cover enforcing and monitoring data contracts using transactional outbox pattern
- Data Contracts: The Mesh Glue Scope of data contract and how it can be a layer with data mesh architectures
- Yet another post on data contracts - part I Thoughts from David J. on what is data contracts and what it should contain
- Yet another post on data contracts - part II What kind of data should be covered by data contracts and some highlights of tooling that’s available
- Yet another post on data contracts - part III Deeper dive on current tooling that can be used for data contracts implementation
- Are we setting data contracts in the right place? Considerations and analogies from software architecture world while defining data contracts
- Implementing data contracts Specifications and real world architecture example of data contract implementation
- The best data contract is the pull request Datafold’s view on how the best data contract is pull request
- An open letter to data ninjas - yes, you need to implement data contracts View on implementations and specifications of data contracts
- Data person: attorney at law Interesting view on data contracts which turn data engineers into data lawyers by Stephen Bailey
- Demystifying event streams - New event streams model at Merit and event specifications being treated as data contracts - named OMG contract
- Fine let’s talk about data contracts - Different view on viability and usability of data contracts
- Brain, kidney, cancer research and data contracts (Schemata project) - Example of data contract explanation / discussion in simple terms for people outside of data space
- The rise of data contracts - (Q/A style article on what are data contracts and why we need them)
- Data contracts from zero to hero - a pragmatic approach to data contracts
- Implementing data contracts - Implementing basic schema based data contract with Kafka schema registry
- Data contracts - ensure robustness in your data mesh architecture - Interesting view on using data sharing agreements in data mesh architecutre and how data contracts validation should be done as data products sidecar
- Data contracts wrapped 2022 - Data contracts creation, storage, enforcement and application
- McDonald's event driven architecture and data journey - Use of data contracts in McDonald's event driven architecture
- Implementing data contracts in data warehouse - Data contracts validation in data warehouse and downstream systems and not just limited to source systems.
- 7 Lessons From GoCardless’ Implementation of Data Contracts
- Data Contracts: The Key to Scaling Distributed Data Architecture and Reducing Data Chaos
- Using Data Contracts with Confluent Schema Registry
- Data Contracts in Action - Testing
Books
Videos
- Inventing data contracts and building data for a FinTech unicorn with Andrew Jones of GoCardless
- Data contracts and domain ownership
- How data contracts can kill innovation
- Getting jiggy with jsonschema: The power of contracts for building data systems
- Data Contracts: Accountable Data Quality w/ Chad Sanderson - UDEM October 2022
- Data Contract Battle Royale w/ Chad Sanderson vs Ethan Aaron
- Driving ML data quality with data contracts
- How to use Data Contracts for Long-Term Schema Management
Podcasts
- Intro to Data Contracts w/ Andrew Jones
- Monday Morning Data Chat (7/3/2023)
- Why You'll Need Data Contracts (w/ Chad Sanderson + Prukalpa)
- The Analytics Engineering Podcast (11/18/2022)
- Data Contracts & Domain Ownership w/ Ananth Packkildurai
- Monday Morning Data Chat (5/24/2022)
- The Importance Of Data Contracts As The Interface For Data Integration With Abhi Sivasailam
- The Data Engineering Podcast (1/23/2022)
Tools
- Avo.app - Avo is a data governance platform that error-proofs your analytics events and speeds up implementation.
- Reflekt - Define tracking plans as code for analytics
- Iteratively - Bring your data teams, product managers and engineers together to define, instrument, verify and collaborate on analytics tracking
- Buz - Buz collects, validates, and delivers schematized data to where it needs to bee
- Benthos - Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.
- Data Caterer - Data Caterer is a test data management tool that can leverage the rich metadata from data contracts to help simulate production-like data and validate your data pipelines
- Provero - Provero is a vendor-neutral, declarative data quality engine with built-in data contracts support. Define checks and contracts in YAML, run anywhere. Features include schema validation, SLA enforcement (freshness, completeness), contract versioning with diff, and violation blocking.
- Schemata - Schemata is a schema modeling framework for decentralized domain-driven ownership of data. Schemata combines a set of standard metadata definitions for each schema & data field and a scoring algorithm to provide a feedback loop on how efficient the data modeling of your data warehouse is.
Specifications and Protocols
- Open Data Contract Specification: A community-driven open standard to define data contract around eight sections: demographics, schema, Data Quality, pricing, stakeholders, security, SLA, and custom properties.
- Data Product Descriptor Specification: An open specification that declaratively defines a data product in all its components using a JSON or YAML descriptor document.
Related
- Establish formal data contracts between consumers and producers Older article from 2018 that explored the concept of data contracts (Older generation article)
- Data Mesh — A Data Movement and Processing Platform @ Netflix - Schema enforcement in Netflix’s data mesh platform
Contributing
Your contributions are most welcome! Please go ahead and create a pull request. We'll review each and every one. We will keep some pull requests open if we not sure whether the PRs are awesome or relevant, you could vote for them by adding 👍 to them. Pull requests will be merged when their votes reach 5.
Contributors
Similar repos
README
Awesome Data Contracts
Data contracts is an amazing initiative to bring data producers and data consumers together. This is a curated list of awesome articles, tools, resources that will help you understand concepts and start your data contracts journey.
Articles and Blogs
- Why the Need for Standardizing Data Contracts? The genesis of Bitol or why data contracts are an essential part of modern data engineering.
- Data Contract 101 A quick and not-so-dirty introduction to data contracts, focusing on the Open Data Contract Standard (ODCS).
- Improving data quality with data contracts (GoCardless) What are data contracts and GoCardless implementation of data contracts
- Interfaces and Breaking Stuff by Tristan Handy (CEO, DBT) Need of data contracts as data interfaces and analogy with software APIs
- Implementing data contracts at GoCardless Data contract specification examples from GoCardless systems
- 7 key learnings from implementing data contracts (Bar Moses and Andrew Jones) Sample data contract example and learnings from implementations
- A PM's thoughts on data contracts A product manager’s view on data contracts and challenges they can solve
- Engineer’s guide to data contracts part I - entities covers implementing data contracts for entities
- Engineer’s guide to data contracts part II - application events cover enforcing and monitoring data contracts using transactional outbox pattern
- Data Contracts: The Mesh Glue Scope of data contract and how it can be a layer with data mesh architectures
- Yet another post on data contracts - part I Thoughts from David J. on what is data contracts and what it should contain
- Yet another post on data contracts - part II What kind of data should be covered by data contracts and some highlights of tooling that’s available
- Yet another post on data contracts - part III Deeper dive on current tooling that can be used for data contracts implementation
- Are we setting data contracts in the right place? Considerations and analogies from software architecture world while defining data contracts
- Implementing data contracts Specifications and real world architecture example of data contract implementation
- The best data contract is the pull request Datafold’s view on how the best data contract is pull request
- An open letter to data ninjas - yes, you need to implement data contracts View on implementations and specifications of data contracts
- Data person: attorney at law Interesting view on data contracts which turn data engineers into data lawyers by Stephen Bailey
- Demystifying event streams - New event streams model at Merit and event specifications being treated as data contracts - named OMG contract
- Fine let’s talk about data contracts - Different view on viability and usability of data contracts
- Brain, kidney, cancer research and data contracts (Schemata project) - Example of data contract explanation / discussion in simple terms for people outside of data space
- The rise of data contracts - (Q/A style article on what are data contracts and why we need them)
- Data contracts from zero to hero - a pragmatic approach to data contracts
- Implementing data contracts - Implementing basic schema based data contract with Kafka schema registry
- Data contracts - ensure robustness in your data mesh architecture - Interesting view on using data sharing agreements in data mesh architecutre and how data contracts validation should be done as data products sidecar
- Data contracts wrapped 2022 - Data contracts creation, storage, enforcement and application
- McDonald's event driven architecture and data journey - Use of data contracts in McDonald's event driven architecture
- Implementing data contracts in data warehouse - Data contracts validation in data warehouse and downstream systems and not just limited to source systems.
- 7 Lessons From GoCardless’ Implementation of Data Contracts
- Data Contracts: The Key to Scaling Distributed Data Architecture and Reducing Data Chaos
- Using Data Contracts with Confluent Schema Registry
- Data Contracts in Action - Testing
Books
Videos
- Inventing data contracts and building data for a FinTech unicorn with Andrew Jones of GoCardless
- Data contracts and domain ownership
- How data contracts can kill innovation
- Getting jiggy with jsonschema: The power of contracts for building data systems
- Data Contracts: Accountable Data Quality w/ Chad Sanderson - UDEM October 2022
- Data Contract Battle Royale w/ Chad Sanderson vs Ethan Aaron
- Driving ML data quality with data contracts
- How to use Data Contracts for Long-Term Schema Management
Podcasts
- Intro to Data Contracts w/ Andrew Jones
- Monday Morning Data Chat (7/3/2023)
- Why You'll Need Data Contracts (w/ Chad Sanderson + Prukalpa)
- The Analytics Engineering Podcast (11/18/2022)
- Data Contracts & Domain Ownership w/ Ananth Packkildurai
- Monday Morning Data Chat (5/24/2022)
- The Importance Of Data Contracts As The Interface For Data Integration With Abhi Sivasailam
- The Data Engineering Podcast (1/23/2022)
Tools
- Avo.app - Avo is a data governance platform that error-proofs your analytics events and speeds up implementation.
- Reflekt - Define tracking plans as code for analytics
- Iteratively - Bring your data teams, product managers and engineers together to define, instrument, verify and collaborate on analytics tracking
- Buz - Buz collects, validates, and delivers schematized data to where it needs to bee
- Benthos - Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.
- Data Caterer - Data Caterer is a test data management tool that can leverage the rich metadata from data contracts to help simulate production-like data and validate your data pipelines
- Provero - Provero is a vendor-neutral, declarative data quality engine with built-in data contracts support. Define checks and contracts in YAML, run anywhere. Features include schema validation, SLA enforcement (freshness, completeness), contract versioning with diff, and violation blocking.
- Schemata - Schemata is a schema modeling framework for decentralized domain-driven ownership of data. Schemata combines a set of standard metadata definitions for each schema & data field and a scoring algorithm to provide a feedback loop on how efficient the data modeling of your data warehouse is.
Specifications and Protocols
- Open Data Contract Specification: A community-driven open standard to define data contract around eight sections: demographics, schema, Data Quality, pricing, stakeholders, security, SLA, and custom properties.
- Data Product Descriptor Specification: An open specification that declaratively defines a data product in all its components using a JSON or YAML descriptor document.
Related
- Establish formal data contracts between consumers and producers Older article from 2018 that explored the concept of data contracts (Older generation article)
- Data Mesh — A Data Movement and Processing Platform @ Netflix - Schema enforcement in Netflix’s data mesh platform
Contributing
Your contributions are most welcome! Please go ahead and create a pull request. We'll review each and every one. We will keep some pull requests open if we not sure whether the PRs are awesome or relevant, you could vote for them by adding 👍 to them. Pull requests will be merged when their votes reach 5.