The Cluster API driver for Magnum enables OpenStack Magnum, a container +orchestration service, to create and manage Kubernetes clusters using the +Cluster API framework. The Cluster API driver for Magnum leverages the power of +the Cluster API to simplify the deployment and management of Kubernetes clusters +within an OpenStack infrastructure.
+With the Cluster API driver for Magnum, Magnum takes on the responsibility of +creating and maintaining the Cluster API resources. Magnum interacts with the +Cluster API controllers and reconcilers to dynamically provision and manage +Kubernetes clusters.
+Magnum utilizes the capabilities of the Cluster API to define the desired state
+of the Kubernetes clusters using familiar Cluster API resources such as Cluster,
+MachineDeployment, and MachineSet. These resources encapsulate important
+cluster configurations, including the number of control plane and worker nodes,
+their specifications, and other relevant attributes.
By leveraging the Cluster API driver for Magnum, Magnum translates the desired +cluster specifications into Cluster API resources. It creates and manages these +resources, ensuring that the Kubernetes clusters are provisioned and maintained +according to the specified configurations.
+Through this integration, Magnum empowers users to leverage the Cluster API's +declarative and consistent approach to manage their Kubernetes clusters in an +OpenStack environment. By leveraging Magnum's container orchestration +capabilities, users can easily create and scale Kubernetes deployments while +benefiting from the automation and extensibility provided by the Cluster API.
+Here references some awesome Intro and Install blogs:
+Openstack Magnum Cluster API written by Satish Patel.OpenStack Magnum Kubernetes Cluster API driver in Kolla-Ansible written by R0K5T4R.clusterctlIf you'd like to track the progress of a specific cluster from the clusterctl
+perspective, you can run the following command to find out the stack_id of the
+cluster and then use clusterctl describe to get the status of the cluster:
$ export CLUSTER_ID=$(openstack coe cluster show <cluster-name> -f value -c stack_id)
+$ watch -cn1 'clusterctl describe cluster -n magnum-system $CLUSTER_ID --grouping=false --color'
+CREATE_IN_PROGRESS stateWith the Cluster API driver for Magnum, the cluster creation process is +performed by the Cluster API for OpenStack. Due to the logic of how the +controller managers work, the process of creating a cluster is performed in +multiple steps and if a step fails, it will keep retrying until it succeeds.
+Unlike the legacy Heat driver, the Cluster API driver for Magnum does not
+move the state of the cluster to CREATE_FAILED if a step fails. Instead,
+it will keep the cluster in CREATE_IN_PROGRESS state until the cluster is
+successfully created or the cluster is deleted.
If you are experiencing issues with the cluster being stuck in CREATE_IN_PROGRESS
+state, you can follow the steps below to troubleshoot the issue:
Check the Cluster name from the stack_id field in Magnum:
$ openstack coe cluster show <cluster-name> -f value -c stack_id
+Check if the Cluster exists in the Kuberentes cluster using the stack_id:
$ kubectl -n magnum-system get clusters <stack-id>
+Note
+If the cluster exists and it is in Provisioned state, you can skip to
+step 3.
You will need to lookup the OpenStackCluster for the Cluster:
$ kubectl -n magnum-system get openstackclusters -l cluster.x-k8s.io/cluster-name=<stack-id>
+Note
+If the OpenStackCluster shows true for READY, you can skip to
+step 4.
You will have to look at the KubeadmControlPlane for
+ the OpenStackCluster:
$ kubectl -n magnum-system get kubeadmcontrolplanes -l cluster.x-k8s.io/cluster-name=<stack-id>
+If the number of READY nodes does not match the number of REPLICAS,
+ you will need to investigate if the instances are going up by looking at
+ the OpenStackMachine for the KubeadmControlPlane:
$ kubectl -n magnum-system describe openstackmachines -l cluster.x-k8s.io/control-plane=,cluster.x-k8s.io/cluster-name=<stack-id>
+From the output, you will need to look at the Status field and see if
+any of the conditions are False. If they are, you will need to look at
+the Message field to see what the error is.
DELETE_IN_PROGRESS stateIf you have a case where a project has been deleted from OpenStack but the +cluster was not deleted, you will not be able to delete it even as an admin +user. You will find log messages such as the following which will indicate +that the project is missing:
+E0705 17:11:00.333902 1 controller.go:326] "Reconciler error" err=<providerClient authentication err: Resource not found: [POST https://cloud.atmosphere.dev/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find project: 1dfcc1f4399948baac7a83a6607f693c.","title":"Not Found"}}
+In order to work around this issue, you will need to create a new project,
+go into the database and update the id of the new project to match the
+project_id of the cluster.
Warning
+It is possible to corrupt the database if you do not know what you are +doing. Please make sure you have a backup of the database before
+Create a new project in OpenStack:
+$ export NEW_PROJECT_ID=$(openstack project create cleanup-project -f value -c id)
+Get the existing project_id of the cluster:
$ export CURRENT_PROJECT_ID=$(openstack coe cluster show <cluster-name> -f value -c project_id)
+Update the id of the project in Keystone to match the project_id of
+ the cluster:
$ mysql -B -N -u root -p -e "update project set id='$CURRENT_PROJECT_ID' where id='$NEW_PROJECT_ID';" keystone
+If you're using Atmosphere, you can run the following:
+$ kubectl -n openstack exec -it sts/percona-xtradb-pxc -- mysql -hlocalhost -uroot -p$(kubectl -n openstack get secret/percona-xtradb -ojson | jq -r '.data.root' | base64 --decode) -e "update project set id='$CURRENT_PROJECT_ID' where id='$NEW_PROJECT_ID';" keystone
+Verify that the project now exists under the new id:
$ openstack project show $CURRENT_PROJECT_ID
+Give access to your current admin user to the new project:
+$ openstack role add --user $OS_USERNAME --project $CURRENT_PROJECT_ID member
+Switch to the context of that user
+$ export OS_PROJECT_ID=$CURRENT_PROJECT_ID
+Create a new set of application credentials and update the existing
+ cloud-config secret for the cluster
$ export CAPI_CLUSTER_NAME=$(openstack coe cluster show tst1-useg-k8s-1 -f value -c stack_id)
+$ export EXISTING_APPCRED_ID=$(kubectl -n magnum-system get secret/$CAPI_CLUSTER_NAME-cloud-config -ojson | jq -r '.data."clouds.yaml"' | base64 --decode | grep application_credential_id | awk '{print $2}')
+$ export EXISTING_APPCRED_SECRET=$(kubectl -n magnum-system get secret/$CAPI_CLUSTER_NAME-cloud-config -ojson | jq -r '.data."clouds.yaml"' | base64 --decode | grep application_credential_secret | awk '{print $2}')
+$ export NEW_APPCRED_ID=$(openstack application credential create --secret $EXISTING_APPCRED_SECRET $CAPI_CLUSTER_NAME-cleanup -f value -c id)
+$ > /tmp/clouds.yaml
+$ kubectl -n magnum-system patch secret/$CAPI_CLUSTER_NAME-cloud-config -p '{"data":{"clouds.yaml":"'$(kubectl -n magnum-system get secret/$CAPI_CLUSTER_NAME-cloud-config -ojson | jq -r '.data."clouds.yaml"' | base64 --decode | sed "s/$EXISTING_APPCRED_ID/$NEW_APPCRED_ID/" | base64 --wrap=0)'"}}'
+At this point, the cluster should start progressing on the deletion process, you +can verify this by running:
+$ kubectl -n capo-system logs deploy/capo-controller-manager -f
+Once the cluster is gone, you can clean up the project:
+$ unset OS_PROJECT_ID
+$ openstack project delete $CURRENT_PROJECT_ID
+The Cluster API driver for Magnum makes use of the Cluster topology feature of the Cluster API project. This allows it to delegate all of the work around building resources such as the OpenStackCluster, MachineDeployments and everything else managed entire by the Cluster API instead of the driver creating all of these resources.
In order to do this, the driver creates a ClusterClass resource which is called magnum-v{VERSION} where {VERSION} is the current version of the driver because of the following reasons:
ClusterClass because it is an immutable resource.ClusterClass.It's important to note that there are only one scenarios where the spec.topology.class for a given Cluster will be modified and this will be when a cluster upgrade is done. This is because there is an expectation by the user that a rolling restart operation will occur if a cluster upgrade is requested. No other action should be allowed to change the spec.topology.class of a Cluster.
For users, it's important to keep in mind that if they want to use a newer ClusterClass in order to make sure of a new feature available in a newer ClusterClass, they can simply do an upgrade within Magnum to the same cluster template and it will actually force an update of the spec.topology.class, which might then naturally cause a full rollout to occur.
In order to be able to test and develop the magnum-cluster-api project, you
+will need to have an existing Magnum deployment. You can use the following
+steps to be able to test and develop the project.
./hack/stack.sh
+pushd /tmp
+source /opt/stack/openrc
+export OS_DISTRO=ubuntu # you can change this to "flatcar" if you want to use Flatcar
+for version in v1.24.16 v1.25.12 v1.26.7 v1.27.4; do \
+ [[ "${OS_DISTRO}" == "ubuntu" ]] && IMAGE_NAME="ubuntu-2204-kube-${version}" || IMAGE_NAME="flatcar-kube-${version}"; \
+ curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/${IMAGE_NAME}.qcow2; \
+ openstack image create ${IMAGE_NAME} --disk-format=qcow2 --container-format=bare --property os_distro=${OS_DISTRO} --file=${IMAGE_NAME}.qcow2; \
+ openstack coe cluster template create \
+ --image $(openstack image show ${IMAGE_NAME} -c id -f value) \
+ --external-network public \
+ --dns-nameserver 8.8.8.8 \
+ --master-lb-enabled \
+ --master-flavor m1.medium \
+ --flavor m1.medium \
+ --network-driver calico \
+ --docker-storage-driver overlay2 \
+ --coe kubernetes \
+ --label kube_tag=${version} \
+ k8s-${version};
+done;
+popd
+openstack coe cluster create \
+ --cluster-template k8s-v1.25.12 \
+ --master-count 3 \
+ --node-count 2 \
+ k8s-v1.25.12
+CREATE_COMPLETE state, you can interact with it:eval $(openstack coe cluster config k8s-v1.25.12)
+In Atmosphere environment, Magnum cluster-api is embedded in the Magnum conductor container.
+Therefore, to upgrade Magnum cluster-api in your atmosphere environment, developers have to build a new Magnum container image with the desired revision.
+On the other hand, magnum-cluster-api repository has a Github workflow to build and push Magnum container images which include the latest Magnum cluster-api driver code. So developers can trigger this workflow to get new Magnum images with the latest code.
Once a new release is published, image build workflow is triggered automatically and new container images is published to quay.io/vexxhost/magnum-cluster-api. So there is no need to run images again and just need to upgrade the image ref in Atmosphere deployment code.
+Run the following cmd to update all Magnum image tags in Atmosphere project.
+
earthly +pin-images
+roles/default/vars/main.yaml file. Then you can run ansible-playbook command to deploy/upgrade atmosphere as normal.
+If you want to apply patches merged into main branch but not released yet, you can follow this instruction.
+- First, build new Magnum container images by running image workflow with Push images to Container Registry enabled at https://github.com/vexxhost/magnum-cluster-api/actions/workflows/image.yml.
+- Once the workflow is successfully finished, new images will be pushed to quay.io/vexxhost/magnum-cluster-api. You can get the exact image tag hash value from the workflow log. (note: It will not promote images so need to get the exact image digest hash value.)
+- Update the image tags in roles/default/vars/main.yaml of atmosphere project. https://github.com/vexxhost/atmosphere/blob/c7c0de94112448522abb8973483da82eb5f937a8/roles/defaults/vars/main.yml#L101-L105
+- Run atmosphere playbook again.
Or you can just update the images on-fly using kubectl CLI in your Atmosphere environment once you know the image ref but this is not recommended. +
kubectl set image sts/magnum-conductor magnum-conductor=${IMAGE_REF} magnum-conductor-init=${IMAGE_REF} -n openstack
+kubectl set image deploy/magnum-api magnum-api=${IMAGE_REF} -n openstack
+kubectl set image deploy/magnum-registry registry=${IMAGE_REF} -n openstack
+