Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing break the glass as a principle #40

Closed
wants to merge 7 commits into from
Closed

Introducing break the glass as a principle #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8769c15
Update PRINCIPLES.md
grmhay Oct 12, 2021
469d730
Update GLOSSARY.md
grmhay Oct 12, 2021
564d81e
Update GLOSSARY.md
grmhay Oct 12, 2021
3fc26ec
Update GLOSSARY.md
grmhay Oct 12, 2021
0069b13
Update GLOSSARY.md
grmhay Oct 20, 2021
c996234
Update PRINCIPLES.md
grmhay Oct 20, 2021
d2110b6
Update PRINCIPLES.md
grmhay Oct 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions GLOSSARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,13 @@ This glossary accompanies the [GitOps Principles](./PRINCIPLES.md), and other su
Git, from which GitOps derives its name, is the canonical example used as this state store but any other system that meets these criteria may be used.
In all cases, these state stores must be properly configured and precautions must be taken to comply with requirements set out in the GitOps Principles.

- ## Intermediate State Store
A system for storing a copy of the declarations that are mastered in the State Store. This system's purpose is intended to bridge the gap in availability between that of the State Store and the expected availability to make configuration changes to the Software System. The Intermediate State Store will offer an availability the same as or near enough to that of the users' expectations to update configuration in the Software System.
Where an Intermediate State Store is used, Reconciliation is used between the State Store and the Intermediate State Store and then again between the Intermediate State Store and the Software System.
Comment on lines +43 to +45
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this part has gotten (and will become) much more relevant with people adopting OCI also for things other than container images. Or as @monadic put it:

GitOps is a transaction system with a Git backend and OCI cache

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we moving into leader election / consensus territory?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do you see leader election or consensus palying a role here? There is exactly one SSoT state store and N intermediate state stores. In simple setups you'll have exactly one intermediate state store. In more complex scenarios you might have more than one (e.g. one per environment). Synchronization always happens unidirectional from state store into the intermediate state stores. The intermediate state stores are independent of each other and don't need to be synchronized laterally.

                           +======+         +==============+
                     +---->| iss1 |<--------| gitops agent |
+=============+      |     +======+         +==============+
| state store |------+            
+=============+      |     +======+         +==============+
                     +---->| iss2 |<--------| gitops agent |
                     |     +======+         +==============+
                     |
                     |     +======+         +==============+
                     +---->| iss3 |<--------| gitops agent |
                           +======+         +==============+

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO what's important to point out here is to make clear the requirements the intermediate state store has to satisfy in order for this whole setup to satisfy the GitOps principles:

  1. It must version all the artifacts
  2. Artifacts, once stored, must be immutable
  3. It must retain a version history


- ## Feedback

Open GitOps follows [control-theory](https://en.wikipedia.org/wiki/Control_theory) and operates in a closed-loop. In control theory, feedback represents how previous attempts to apply a desired state have affected the actual state. For example if the desired state requires more resources than exist in a system, the software agent may make attempts to add resources, to automatically rollback to a previous version, or to send alerts to human operators.

- ## Break the Glass
The process of editing the Intermediate State Store directly in the event that a configuration update needs to be made to the Software System but the State Store is unavailable.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suppose an example of why the state store is not available might be helpful to understand the potential situation better. Also a note that this should really only be a very rare exception and that proper authorization must be in place (e.g. multi-party authorization).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

State store can be anything? s3? cassandra?

3 changes: 3 additions & 0 deletions PRINCIPLES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,6 @@ The [desired state](./GLOSSARY.md#desired-state) of a GitOps managed system must

Software agents [continuously](./GLOSSARY.md#continuous) observe actual system state and [attempt to apply](./GLOSSARY.md#reconciliation) the desired state.

5. **Actual System State is Manageable "always"**

System state manageability must meet users' SLA expectations to make changes to it. If the desired state store's availability is less that that SLA expectation, it should be possible to "break the glass" to update the system state directly and reconcile system state to desire state.