cloudogu/gitops-patterns

Intro

This repo collects some patterns (AKA strategies, models, approaches, best practices, etc.) about GitOps process design (AKA GitOps repository structures, etc.) as well as links to talks, articles and example repos.

It originated in schnatterer's talk about the "perfect" GitOps process, our experience in consulting, and our GitOps trainings for both Flux and ArgoCD.

You can read more about the patterns in this series of articles.

PRs welcome!

Contents

• Articles
• Pattern categories
• GitOps Patterns
  • Operator deployment
    • Hub and Spoke
    • Instance per Cluster
    • Instance per Namspace
    • Split-Instance
  • Repository structure
    • Monorepo
    • Repo per Team
    • Repo per Application (can be implemented via Repo Separation, Config Replication, Repo Pointer or Config Split)
    • Repo per Environment
  • Promotion
    • Environments
      • Folder/Directory per environment
      • Repo per environment
      • Branch per environment
      • Preview environments
    • Implementation
      • Configuration Management, includes Rendered Manifests and Umbrella Chart
      • Global Environments vs Environment per App
      • Config update
  • Wiring
    • Bootstrapping
    • Linking, includes Nesting (e.g. App of Apps), Templating
• Public GitOps Repo Examples
  • GitOps Playground
  • ArgoCD autopilot
  • Flux Monorepo
  • Flux repo per team/tenant
  • 📕 Path to GitOps examples
  • Environment variations
  • Multiple AppSets
  • Others
• Synonyms
• References

Articles

Pattern categories

Let's group GitOps patterns into separate categories to make them easier to grasp.
See also part 1️⃣ of the article series.

• Operator deployment: GitOps operators ↔ Clusters/Namespaces
• Repository: How many repos?
• Promotion:
  • Environments: How to model environments/stages?
  • Implementation: How to implement the actual process of promotion?
• Wiring: Bootstrapping operator, linking repos and folders

GitOps Patterns

Operator deployment

See also part 2️⃣ of the article series.

• Hub and Spoke ¹² 1 Operator : n Clusters
  Synonyms: Management Cluster ³ - sounds familiar, e.g. from Cluster API, but it requires a cluster. In SaaS/PaaS constellations, the central instance may not even be operated in a cluster. Therefore, the term "Hub and Spoke" seems more fitting to us, especially since it is also well known from computer networks or logistics.
  
• Instance Per Cluster ³ : 1 Operator : 1 Cluster
  Synonyms: Standalone ¹ - could apply to both "Instance per Cluster" and "Instance per Namespace"; does not define the relationship to clusters and namespaces precisely enough, as not every operator supports both deployment patterns.
  
• Instance per Namespace: n Operators : 1 Cluster
  Synonyms: Namespaced ⁴
  

More Patterns:

• Split-Instance¹: 1 Operator : n Clusters; components split between management and target clusters
• Hybrids of Hub and Spoke and Instance per Cluster patterns: "Instance per Logical Group" and "Argo Managing more Argos".³

Repository structure

See also part 3️⃣ of the article series.

• Monorepo ²⁵
  Opposite: polyrepo ²
• Repo per Team ⁵
  More generalized: Team ➡️ Tenant⁵
• Repo per Application ²⁵
  Implementations:
  • Repo separation ⁶
    Keep code in app repo, config in config repo
    
  • Config replication ⁴
    Keep config in app repo and have CI server replicate it to the config repo
    
  • Repo pointer ⁷⁴⁵
    Keep the whole config in app repo and add a pointer from config repo (e.g. Argo CD Application or a Flux GitRepository+Kustomization), avoiding redundancy
    
  • Config Split ⁸
    Keep parts of the config in app repo (e.g. helm chart or kustomize base), and rest in config repo (e.g. value.yamls or overlays for different envs).
    Then have the config repo point to the app repo (e.g. via ArgoCD Application or Flux Kustomization).
    
    Alternative implementations: have CI server push chart to helm registry
    
    Or use push helm chart to OCI registry
    
    Or use any config management tool (e.g. helm template, kustomize build, kubectl kustomize, jsonnet, cuelang, timoni, etc. ) on the CI server for pushing the final manifests as OCI artifacts⁹ to the registry (Rendered Manifests Pattern¹⁰).
    Then have the config repo point to the OCI artifact (e.g. via Flux Kustomization).
    This way, the OCI registry functions as a "GitOps Cache"¹¹:
    The operator only needs to pull the artifacts instead of rendering/overlaying the config from different sources.
    
• Repo per environment ⁵
  Synonym: Environment per repository¹², Repo per Stage

Promotion

See also part 4️⃣ of the article series.

We understand the term "promotion" as the process of deploying applications to different environments.
Sometimes, the term "promotion" is prefixed with other words: Release/Application/Environment/Workload/Change.

For promotion, we see different sets of patterns:

• one regarding the modelling of environments and
• one regarding the implementation of the actual process of promotion.

Environments

• Folder/Directory per environment
  Synonym: Environment per folder ¹²
• Repo per environment ⁵
  Synonym: Environment per repo ¹²
• Branch per environment
  Synonym: Environment per branch ¹²
  Often discouraged ⁷ or declared an anti-pattern¹³¹⁴, but can work ¹⁵.
• Preview environments ^{16 17 18}
  Synonyms: ephemeral, dynamic, pull request¹⁹, test, temporary¹⁶
  Beyond the GitOps world also known as "Preview Deployments"²⁰ and "Deploy Previews"²¹

Implementation

• Configuration Management
  • Synonyms:
    • General: Templating, Patching, Overlay, Rendering, Hydrating
    • templates = dry; rendered = hydrated ²²
  • Where to render the manifests?
    • On the CI Server (Rendered Manifests Pattern¹⁰)
      This pattern also fits nicely with Flux's OCI artifacts feature (see Config split).
    • On the GitOps Operator (GitOps Operator Rendering or GitOps Controller Rendering²³)
  • Tools:
    • Kustomize
      • Plainkustomization.yaml - operator-agnostic (GitOps Operator Rendering on Argo CD and Flux)
      • ≠ Flux CRD Kustomization (GitOps Operator Rendering)
      • kustomize build / kubectl kustomize via CI server (Rendered Manifests)
    • Helm
      • CRD such as HelmRelease (GitOps Operator Rendering on Flux) or Application (GitOps Operator Rendering on ArgoCD)
      • Umbrella Chart²⁴ (common on Argo CD, possible but unusual on Flux)
      • helm template on CI server (Rendered Manifests)
    • Others, e.g. Jsonnet, CueLang, Timoni
      • Rendered Manifests Pattern
      • Argo CD Config Management Plugins²⁵ (built-in for JSonnet)
• Global Environments vs Environment per App⁴
  
• Config update
  Who updates image (version) in GitOps repo, creates branch and PR?
  • Manual: Human pushes branch and create PR 🥵
  • CI Server: Build job pushes branch, creates PR
    
  • Image Updater: Operator pushes branch, create PR manually
    
  • Dependency Bot: Bot pushes branch, creates PR
    

Wiring

See also part 5️⃣ of the article series.

• Bootstrapping , e.g. using kubectl or operator-specific CLI such as flux or argocd-autopilot
• Linking e.g. using Operator-specific CRDs such as Kustomization (Flux) or Application (ArgoCD)
  • Nesting, e.g. App of Apps pattern ²⁶ (ArgoCD)
  • Templating, e.g. implemented using ApplicationSets

Public GitOps Repo Examples

See also ⁴.

See also part 6️⃣ of the article series.

The diagrams have been created with tree2svg.

GitOps Playground

cloudogu/gitops-playground

• Repo pattern: Per team mixed with per app (implemented via Config Replication)
• Operator pattern: Instance per Cluster (Hub and Spoke also possible)
• Operator: ArgoCD (Flux)
• Boostrapping: Helm, kubectl
• Linking: ArgoCD Application
• Features:
  • Operate ArgoCD with GitOps
  • Solution for cross-cutting infra (cluster resources)
  • Multi-tenancy: Single operator manages multiple tenants on one cluster using namespace environments (multi-cluster possible)
  • Mixed repo patterns
  • Config Update and Config Replication via CI server (Rendered Manifest using Helm)
  • Env per app Pattern

ArgoCD autopilot

argoproj-labs/argocd-autopilot

• Repo pattern: Monorepo
• Operator pattern: Instance per Cluster / Hub and Spoke
• Operator: ArgoCD
• Boostrapping: argocd-autopilot
• Linking: kustomization.yaml, ArgoCD Application, ApplicationSet
• Features:
  • Create structure and YAML via CLI
  • Operate ArgoCD with GitOps
  • Solution for cross-cutting infra (cluster resources)
  • Env per app Pattern

Flux Monorepo

fluxcd/flux2-kustomize-helm-example

• Repo pattern: Monorepo
• Operator pattern: Instance per Cluster
• Operator: Flux
• Boostrapping: flux CLI
• Linking: kustomization.yaml, Flux Kustomization
• Features:
  • Operate Flux with GitOps
  • Solution for cross-cutting infra (cluster resources)

Flux repo per team/tenant

fluxcd/flux2-multi-tenancy

• Repo pattern: Repo per team/tenant
• Operator pattern: Instance per Cluster
• Operator: Flux
• Boostrapping: flux CLI
• Linking: kustomization.yaml, Flux Kustomization
• Features:
  • Operate Flux with GitOps
  • Solution for cross-cutting infra (cluster resources)
  • Multi-tenancy: One operator per cluster environment manages multiple tenants

Flux D1 Reference Architecture

To be discussed in more detail

• controlplaneio-fluxcd/d1-infra
• PDF

📕 Path to GitOps examples

christianh814/example-kubernetes-go-repo

christianh814/example-kubernetes-goflux-repo

christianh814/example-openshift-go-repo

• Repo pattern: Monorepo
• Operator pattern: Instance per Cluster
• Operator: Argo CD, Flux
• Boostrapping: kubectl
• Linking: kustomization.yaml,
  • ArgoCD Application, ApplicationSet /
  • Flux Kustomization
• Features:
  • Cross-cutting infra and app(s)
  • ArgoCD and Flux examples
  • Env per app Pattern

Environment variations

kostis-codefresh/gitops-environment-promotion

• Operator: ArgoCD (Flux)
• Linking: kustomization.yaml
• Features:
  • Env variants for single app
  • Promotion "via cp"
  • Env per app Pattern

Multiple AppSets

kostis-codefresh/many-appsets-demo

• Operator: Argo CD
• Linking: kustomization.yaml, Argo CD ApplicationSet, (Application)
• Features:
  • Env variants for multiple apps with few appSets
  • Promotion "via cp"
  • Env per app Pattern

Others

Here are some other examples that we haven't had a chance to look at in more detail:

• onedr0p/cluster-template (example: onedr0p/home-ops- A famous mono repo that hosts everything to boostrap a plain k8s cluster with many cluster resources using Flux
• Lp-Francois/argocd-multi-tenancy

Synonyms

• Patterns ≈ strategies, models, approaches, best practices, standards
• GitOps process design ≈ GitOps repository structures,
• GitOps Operator ≈ GitOps controller ≈ GitOps agent
• Config Repo = GitOps repo, Infra repo, Payload repo
  
• App repo = Source code repo, Source repo
  
• Environment = Stage
• Folder = Directory
• Templating ≈ Patching, Overlay, Rendering, Bundling, Packaging?

References

Footnotes

1. Article A Comprehensive Overview of Argo CD Architectures – 2023 by Dan Garfield ↩ ↩² ↩³

2. Article/Book How to set up your GitOps directory structure by Christian Hernandez ↩ ↩² ↩³ ↩⁴

3. Talk Control Plane, Service, or Both? – Argo CD Multi-Cluster Architectures - Nicholas Morey, Akuity, Article How many do you need? - Argo CD Architectures Explained by Nicholas Morey ↩ ↩² ↩³

4. Slides The perfect GitOps process: repos, folders, stages, patterns by Johannes Schnatterer ↩ ↩² ↩³ ↩⁴ ↩⁵

5. Documentation Flux | Ways of structuring your repositories ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

6. Documentation Argo CD: Best Practices ↩

7. Talk GitOps: Core Concepts & Ways of Structuring Your Repos by Pinky Ravi and Scott Rigby ↩ ↩²

8. Discussion on LinkedIn Benjamin Ruland and Johannes Schnatterer ↩

9. Documentation Flux | OCI cheatsheet ↩

10. Talk: GitOpsCon EU 23: The Art of GitOps: Rendered Manifests by Christian Hernandez ↩ ↩²

11. Talk: Mastering GitOps 2023: Keynote: GitOps Emerging Developments and Predictions by Alexis Richardson ↩

12. Lesson GitOps at Scale Lesson series - Git repository strategies by Codefresh (paywalled) ↩ ↩² ↩³ ↩⁴

13. Article Stop Using Branches for Deploying to Different GitOps Environments by Kostis Kapelonis ↩

14. Article Git best practices: Workflows for GitOps deployments by Christian Hernandez ↩

15. Article Monitoring and Hardening the GitOps Delivery Pipeline with Flux by Florian Heubeck ↩

16. Article Creating Temporary Preview Environments Based On Pull Requests With Argo CD And Codefresh by Codefresh ↩ ↩²

17. Talk GitOps Con Europe - Implementing Preview Environments with GitOps in Kubernetes by François Le Pape, Remazing ↩

18. Talk Preview Environments with ArgoCD by Brandon Phillips ↩

19. Video Environments Based On Pull Requests (PRs): Using Argo CD To Apply GitOps Principles On Previews by Viktor Farcic ↩

20. Documentation Vercel: Preview Deployments ↩

21. Documentation Netlify: Deploy Previews ↩

22. Discussion on GitHub Michael Crenshaw ↩

23. Discussion on LinkedIn Christian Hernandez, Scott Rigby and Johannes Schnatterer ↩

24. Documentation Helm | Chart Development Tips and Tricks ↩

25. Documentation Argo CD | Config Management Plugins ↩

26. Documentation ArgoCD: Cluster Bootstrapping - App Of Apps Pattern ↩