Merge pull request #1033 from openziti/ha-docs-bootstrap

Add controller clustering bootstrapping doc

docusaurus/docs/reference/ha/_category_.yml

@@ -1,5 +1,2 @@
-label: Controller HA
+label: Controller Clustering
 position: 22
-link:
-  type: doc
-  id: reference/ha/overview

docusaurus/docs/reference/ha/bootstrapping/_category_.yml

@@ -0,0 +1,6 @@
+label: Bootstrapping
+position: 10
+link:
+  type: doc
+  id: reference/ha/bootstrapping/overview
+

docusaurus/docs/reference/ha/bootstrapping/certificates.md

@@ -0,0 +1,84 @@
+---
+sidebar_label: Certificates
+sidebar_position: 20
+---
+
+# Controller Certificates
+
+For controllers to communicate and trust one another, they need certificates that have
+been generated with the correct attribute and relationships.
+
+## Requirements
+
+1. The certificates must have a shared root of trust
+2. The controller client and server certificates must contain a 
+   [SPIFFE ID](https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/#spiffe-id)
+
+## Steps to Certificate Creation
+There are many ways to set up certificates, so this will just cover a recommended configuration.
+
+The primary thing to ensure is that controllers have a shared root of trust. 
+A standard way of generating certs would be as follows:
+
+1. Create a self-signed root CA
+1. Create an intermediate CA for each controller
+1. Issue a server cert using the intermediate CA for each controller
+1. Issue a client cert using the intermediate CA for each controller
+
+Note that controller server and client certs must contain a SPIFFE id of the form
+
+```
+spiffe://<trust domain>/controller/<controller id>
+```
+
+So if your trust domain is `example.com` and your controller id is `ctrl1`, then your SPIFFE id
+would be:
+
+```
+spiffe://example.com/controller/ctrl1
+```
+
+**SPIFFE ID Notes:**
+
+* This ID must be set as the only URI in the `X509v3 Subject Alternative Name` field in the
+  certificate.
+* These IDs are used to allow the controllers to identify each during the mTLS negotiation.
+* The OpenZiti CLI supports creating SPIFFE IDs in your certs
+    * Use the `--trust-domain` flag when creating CAs
+    * Use the `--spiffe-id` flag when creating server or client certificates
+
+## Example
+
+Using the OpenZiti PKI tool, certificates for a three node cluster could be created as follows:
+
+```bash
+# Create the trust root, a self-signed CA
+ziti pki create ca --trust-domain ha.test --pki-root ./pki --ca-file ca --ca-name 'HA Example Trust Root'
+
+# Create the controller 1 intermediate/signing cert
+ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl1 --intermediate-name 'Controller One Signing Cert'
+
+# Create the controller 1 server cert
+ziti pki create server --pki-root ./pki --ca-name ctrl1 --dns "localhost,ctrl1.ziti.example.com" --ip "127.0.0.1,::1" --server-name ctrl1 --spiffe-id 'controller/ctrl1'
+
+# Create the controller 1 server cert
+ziti pki create client --pki-root ./pki --ca-name ctrl1 --client-name ctrl1 --spiffe-id 'controller/ctrl1'
+
+# Create the controller 2 intermediate/signing cert
+ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl2 --intermediate-name 'Controller Two Signing Cert'
+
+# Create the controller 2 server cert
+ziti pki create server --pki-root ./pki --ca-name ctrl2 --dns "localhost,ctrl2.ziti.example.com" --ip "127.0.0.1,::1" --server-name ctrl2 --spiffe-id 'controller/ctrl2'
+
+# Create the controller 2 client cert
+ziti pki create client --pki-root ./pki --ca-name ctrl2 --client-name ctrl2 --spiffe-id 'controller/ctrl2'
+
+# Create the controller 3 intermediate/signing cert
+ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl3 --intermediate-name 'Controller Three Signing Cert'
+
+# Create the controller 3 server cert
+ziti pki create server --pki-root ./pki --ca-name ctrl3 --dns "localhost,ctrl3.ziti.example.com" --ip "127.0.0.1,::1" --server-name ctrl3 --spiffe-id 'controller/ctrl3'
+
+# Create the controller 3 client cert
+ziti pki create client --pki-root ./pki --ca-name ctrl3 --client-name ctrl3 --spiffe-id 'controller/ctrl3'
+```

docusaurus/docs/reference/ha/bootstrapping/configuration.md

@@ -0,0 +1,51 @@
+---
+sidebar_label: Configuration
+sidebar_position: 30
+---
+
+# Controller Configuration
+
+### Config File
+
+The controller requires a `cluster` section.
+
+```yaml
+cluster:
+  dataDir: /path/to/data/dir
+```
+
+The `dataDir` will be used to store the following:
+
+* `ctrl-ha.db` - the OpenZiti data model bbolt database
+* `raft.db` - the Raft bbolt database
+* `snapshots/` - a directory to store Raft snapshots
+
+Controllers use the control channel listener to communicate with each other. Unlike
+routers, they need to know how to reach each other, so an advertise address must
+be configured.
+
+```yaml
+ctrl:
+  listener: tls:0.0.0.0:6262
+  options:
+    advertiseAddress: tls:ctrl1.ziti.example.com:6262
+```
+
+Finally, cluster-capable SDK clients use OIDC for authentication, so an OIDC endpoint must be configured.
+
+```yaml
+web:
+  - name: all-apis-localhost
+    bindPoints:
+      - interface: 0.0.0.0:1280
+        address: ctrl1.ziti.example.com:1280
+    options:
+      minTLSVersion: TLS1.2
+      maxTLSVersion: TLS1.3
+    apis:
+      - binding: health-checks
+      - binding: fabric
+      - binding: edge-management
+      - binding: edge-client
+      - binding: edge-oidc
+```

docusaurus/docs/reference/ha/bootstrapping/initialization.md

@@ -0,0 +1,37 @@
+---
+sidebar_label: Initialization
+sidebar_position: 40
+---
+
+# Initializing the First Controller
+
+First start the controller:
+
+```shell
+ziti controller run </path/to/controller-config.yml>
+```
+
+Since this controller has not yet been initialized, it does not have an administrator 
+identity that can be used to manage the network. The controller will pause startup
+and wait for initialization. While waiting it will periodically emit a message:
+
+```buttonless title="Output"
+[   3.323] WARNING ziti/controller/server.(*Controller).checkEdgeInitialized: the 
+Ziti Edge has not been initialized, no default admin exists.  Add this node to a 
+cluster using 'ziti agent cluster add tls:localhost:6262' against an existing 
+cluster member, or if this is the bootstrap node, run 'ziti agent controller init' 
+to configure the default admin and bootstrap the cluster
+```
+
+As this is the first node in the cluster, there's no existing cluster to add it to. 
+
+To add the default administrator, run:
+
+```
+ziti agent controller init <admin username> <admin password> <admin identity name>
+```
+
+This initializes an admin user that can be used to manage the network.
+
+Once the admin user is created, the controller should be up and running. This is
+now a functional HA cluster, albeit with a cluster size of one.

docusaurus/docs/reference/ha/bootstrapping/overview.md

@@ -0,0 +1,14 @@
+---
+sidebar_label: Overview
+sidebar_position: 10
+---
+
+# Bootstrapping A Cluster 
+
+To bring up a controller cluster, one starts with a single node. 
+
+Bootstrapping a cluster has the following steps:
+
+1. [Creating Certificates](certificates.md)
+1. [Setting Controller Config](configuration.md)
+1. [Controller Initialization](initialization.md)

docusaurus/docs/reference/ha/overview.md

@@ -1,4 +1,9 @@
-# Controller HA 
+---
+sidebar_label: Overview
+sidebar_position: 5
+---
+
+# Controller Clustering
 
 ## Overview