Skip to content

Commit

Permalink
OPENSHIFTP-201: Pull Request created for CICD-pipeline with sh file c…
Browse files Browse the repository at this point in the history 
…hanges and yaml file changes. (#70)

* Changes checked in for 201. Initial Draft of README.md; etcd-mc.yaml and
mount_etcd_to_ext_vol.sh

Changes incorporated as per the review comments for etcd-mc.yaml and
mount_etcd_to_ext_vol.sh.
etcd-mc.yaml --> spec version updated to 3.2; Extra garbage lines removed
hani
mount_etct_to_ext_vol.sh -->
1. var_rg updated
2. ibmcloud replaced with ${IBMCLOUD}; corresponding code block added.
3. rdr-ca referred as var_vpc_prefix
4. While loop added to check for Volume status. While loop will run
   until Volume status is other than available(Pending).
5. If stmt added for CICD variable. We will mention "We do not backup
   etcd" for CICD.
6. Need help with the next step --> Execute the MachineConfig to
   migrate/attach. We need to do oc login before executing the command "oc create
   -f etcd-mc.yaml".

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>

etcd-mc.yaml --> spec version updated to 3.2; Extra garbage lines removed
README.md --> Lines added for exit code 77
mount_etct_to_ext_vol.sh -->
1. var_rg updated
2. ibmcloud replaced with ${IBMCLOUD}; corresponding code block added.
3. rdr-ca referred as var_vpc_prefix
4. While loop added to check for Volume status. While loop will run
   until Volume status is other than available(Pending).
5. If stmt added for CICD variable. We will mention "We do not backup
   etcd" for CICD.
6. Need help with the next step --> Execute the MachineConfig to
   migrate/attach. We need to do oc login before executing the command "oc create
   -f etcd-mc.yaml".
Commit with Amend option.

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>

Changes done for Issue#201(https://jsw.ibm.com/browse/OPENSHIFTP-201)

Changes incorporated as per the review comments for etcd-mc.yaml and
mount_etcd_to_ext_vol.sh.
etcd-mc.yaml --> spec version updated to 3.2; Extra garbage lines removed
README.md --> Lines added for exit code 77
mount_etct_to_ext_vol.sh -->
1. var_rg updated
2. ibmcloud replaced with ${IBMCLOUD}; corresponding code block added.
3. rdr-ca referred as var_vpc_prefix
4. While loop added to check for Volume status. While loop will run
   until Volume status is other than available(Pending).
5. If stmt added for CICD variable. We will mention "We do not backup
   etcd" for CICD.
6. Need help with the next step --> Execute the MachineConfig to
   migrate/attach. We need to do oc login before executing the command "oc create
   -f etcd-mc.yaml".

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>

etcd-mc.yaml --> spec version updated to 3.2; Extra garbage lines removed
README.md --> Lines added for exit code 77
mount_etct_to_ext_vol.sh -->
1. var_rg updated
2. ibmcloud replaced with ${IBMCLOUD}; corresponding code block added.
3. rdr-ca referred as var_vpc_prefix
4. While loop added to check for Volume status. While loop will run
   until Volume status is other than available(Pending).
5. If stmt added for CICD variable. We will mention "We do not backup
   etcd" for CICD.
6. Need help with the next step --> Execute the MachineConfig to
   migrate/attach. We need to do oc login before executing the command "oc create
   -f etcd-mc.yaml".
Commit with Amend option.

Shell script added for Mounting external volume and copying etcd contents to external volume

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>

Changes done for the script file:/mount_etcd_ext_volume.sh ( Creating
and attaching the volume;  running the machineconfig yaml file and then
also verifying if the mount point is correctly created)

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>

* Update scripts/etcd-mount/ocp_login.sh

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* Update scripts/etcd-mount/mount_etcd_ext_volume.sh

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

---------

Signed-off-by: mpkpersistent <130049908+mpkpersistent@users.noreply.github.com>
Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
Co-authored-by: Paul Bastide <paul@bastide.org>
  • Loading branch information
@mpkpersistent @prb112
mpkpersistent and prb112 authored Sep 10, 2024
1 parent d167208 commit 67ed86e
Show file tree
Hide file tree
Showing 4 changed files with 225 additions and 0 deletions.
88 changes: 88 additions & 0 deletions scripts/etcd-mount/98-master-lib-etcd-mc.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 98-var-lib-etcd
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: /etc/find-secondary-device
mode: 493
contents:
source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKc2V0IC11byBwaXBlZmFpbAoKZm9yIGRldmljZSBpbiAvZGV2L3ZkKjsgZG8KL3Vzci9zYmluL2Jsa2lkICRkZXZpY2UgJj4gL2Rldi9udWxsCmlmIFsgJD8gPT0gMiBdOyB0aGVuCiAgZWNobyAic2Vjb25kYXJ5IGRldmljZSBmb3VuZCAkZGV2aWNlIgogIGVjaG8gImNyZWF0aW5nIGZpbGVzeXN0ZW0gZm9yIGV0Y2QgbW91bnQiCiAgbWtmcy54ZnMgLUwgdmFyLWxpYi1ldGNkIC1mICRkZXZpY2UgJj4gL2Rldi9udWxsCiAgdWRldmFkbSBzZXR0bGUKICB0b3VjaCAvZXRjL3Zhci1saWItZXRjZC1tb3VudAogIGV4aXQKZmkKZG9uZQplY2hvICJDb3VsZG4ndCBmaW5kIHNlY29uZGFyeSBibG9jayBkZXZpY2UhIiA+JjIKZXhpdCA3Nwo=
systemd:
units:
- name: find-secondary-device.service
enabled: true
contents: |
[Unit]
Description=Find secondary device
DefaultDependencies=false
After=systemd-udev-settle.service
Before=local-fs-pre.target
ConditionPathExists=!/etc/var-lib-etcd-mount
[Service]
RemainAfterExit=yes
ExecStart=/etc/find-secondary-device
RestartForceExitStatus=77
[Install]
WantedBy=multi-user.target
- name: var-lib-etcd.mount
enabled: true
contents: |
[Unit]
Before=local-fs.target
[Mount]
What=/dev/disk/by-label/var-lib-etcd
Where=/var/lib/etcd
Type=xfs
TimeoutSec=120s
[Install]
RequiredBy=local-fs.target
- name: sync-var-lib-etcd-to-etcd.service
enabled: true
contents: |
[Unit]
Description=Sync etcd data if new mount is empty
DefaultDependencies=no
After=var-lib-etcd.mount var.mount
Before=crio.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
ExecStart=/usr/sbin/setsebool -P rsync_full_access 1
ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
ExecStart=/usr/sbin/semanage fcontext -a -t container_var_lib_t '/var/lib/etcd(/.*)?'
ExecStart=/usr/sbin/setsebool -P rsync_full_access 0
TimeoutSec=0
[Install]
WantedBy=multi-user.target graphical.target
- name: restorecon-var-lib-etcd.service
enabled: true
contents: |
[Unit]
Description=Restore recursive SELinux security contexts
DefaultDependencies=no
After=var-lib-etcd.mount
Before=crio.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/restorecon -R /var/lib/etcd/
TimeoutSec=0
[Install]
WantedBy=multi-user.target graphical.target
32 changes: 32 additions & 0 deletions scripts/etcd-mount/README.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,33 @@
This automation is used in CI/CD only. It is not intended for production use.

Automation : Moving etcd to an attached volume.

Pre-requisites:
This script requires that external block volume of following specification is attached to the Master node of OCP Cluster.
Size : Min 20 GB.
Profile : 5iops-tier
Auto-Delete : True

Note for using Virtual Device:
We used the External block volumes for testing purpose and found that it is attached as Virtual Device.
Hence the script mentions /dev/vd*.
Please refer article for other options of external drives.(SCSI/SATA/NVM)

Artifacts:
Script : mount_etcd_ext_volume.sh
MachineConfig.yaml : 98-master-lib-etcd-mc.yaml

Input Parameters:
Tier : Can be either "10iops-tier" or "5iops-tier"
Resource group: Will be specific to Customer(ocp-dev-resource-group in our case)
Tag : Optional
VPC Prefix : Used by the Customer


Steps to run Automation:
1. CI/CD will take care of backing up the existing etcd data.
2. CI/CD will add the MachineConfig Yaml file into the OCP Cluster and the MachineConfig CR will move the etcd to the attached block volume.
3. If the logs show an exit code of 77, then a failure moving the etcd data in /etc/find-secondary-device happened.

Please refer the article for more details:
https://docs.openshift.com/container-platform/4.16/scalability_and_performance/recommended-performance-scale-practices/recommended-etcd-practices.html#move-etcd-different-disk_recommended-etcd-practices
104 changes: 104 additions & 0 deletions scripts/etcd-mount/mount_etcd_ext_volume.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
################################################################
# Copyright 2024 - IBM Corporation. All rights reserved
# SPDX-License-Identifier: Apache-2.0
#################################################################


#!/bin/bash
set -o errexit
set -uo pipefail


IBMCLOUD=ibmcloud
IBMCLOUD_HOME_FOLDER=""
CICD=""

if [ -z "${CICD}" ]
then
echo "We do not backup etcd"
#exit 0
fi

sh ocp_login.sh

if [[ $(type -t ic) == function ]]
then
IBMCLOUD=ic
else
${IBMCLOUD} plugin install power-iaas -f
fi

if [ ! -z "${IBMCLOUD_HOME_FOLDER}" ]
then
IBMCLOUD_HOME_FOLDER="${1}"
function ic() {
HOME=${IBMCLOUD_HOME_FOLDER} ibmcloud "$@"
}
IBMCLOUD=ic
fi


#var_tier="10iops-tier"
#var_rg="ocp-dev-resource-group"
#var_rg="${ENV_RESOURCE_GROUP:-ocp-dev-resource-group}"
#var_tag="rdr-multi-arch-etcd"
var_rand_id=$(echo "$(openssl rand -hex 4)");
#var_vpc_prefix=rdr-mac-mkul18
vsi_out=$(${IBMCLOUD} is instances | grep ${var_vpc_prefix} | grep master | awk -vOFS=":" '{print $1,$2,$9}');
#echo $vsi_out;
arr=( $(sed 's/:/ /g' <<<"$vsi_out") )
i=0;
for count in 0 1 2; do
id=${arr[i]};
name=${arr[i+1]};
region=${arr[i+2]};
vol_create_command="${IBMCLOUD} is volume-create auto-etcd-vol-${var_rand_id}-${count} ${var_tier} ${region} --capacity 20 --resource-group-name ${var_rg} --output JSON --tags ${var_tag}"
# echo ${vol_create_command};
VOLUME_ID=$(${IBMCLOUD} is volume-create auto-etcd-vol-${var_rand_id}-${count} ${var_tier} ${region} --capacity 20 --resource-group-name ${var_rg} --output JSON --tags ${var_tag} | jq .id | tr -d "'\"")
VOL_STATUS=$( ${IBMCLOUD} is volumes | grep ${VOLUME_ID} | awk '{print $3}' );
while [ "$VOL_STATUS" != "available" ]
do
VOL_STATUS=$( ${IBMCLOUD} is volumes | grep ${VOLUME_ID} | awk '{print $3}' );
done

vol_attach_command="${IBMCLOUD} is instance-volume-attachment-add auto-attach-vol${count} ${id} ${VOLUME_ID} --auto-delete true --output JSON --tags ${var_tag}"
echo ${vol_attach_command};
ATTACH_COMMAND=$(${IBMCLOUD} is instance-volume-attachment-add auto-attach-vol${count} ${id} ${VOLUME_ID} --auto-delete true --output JSON --tags ${var_tag});
echo "Volume Attached Successfully to the Master Node : ${name}"
echo "Waiting while the attachment is activated"
sleep 10
chk_query="${IBMCLOUD} is instance-volume-attachments ${name}"
echo ${chk_query}
if [ -z "$(ibmcloud is instance-volume-attachments ${name} --output json | jq -r '.[] | select(.status != "attached")')" ]
then
echo "Delaying as not all volumes are finished attaching to instance"
sleep 60
fi
i=$((i+3));
done

#Volume Attachment done. Time for etcd migration
sleep 30s
echo "Going for mc-update-file"
oc apply -f 98-master-lib-etcd-mc.yaml --kubeconfig=auth/kubeconfig
# adding a sleep 30 here is not harmful or causing unnecessary delays.
sleep 30
echo "Waiting on the mcp/master to update"
oc wait --for=condition=updated mcp/master --timeout=50m --kubeconfig=auth/kubeconfig

echo "etcd migration done successfully."
#etcd migration done Verification start
i=0;
for count in 0 1 2; do
name=${arr[i+1]}
echo "Logging inside Node : ${name}"
mount_out=$(oc debug --as-root=true node/$name -- chroot /host grep -w "/var/lib/etcd" /proc/mounts)
echo "Mountpoint : ${mount_out} "
if [[ ${mount_out} = "" ]]; then
echo "etcd mount fail for : ${name}"
exit 0
fi
i=$((i+3))
done

echo "Done Mounting"
1 change: 1 addition & 0 deletions scripts/etcd-mount/ocp_login.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
oc login -s https://api.<domain>:6443 -u kubeadmin -p <Password>

0 comments on commit 67ed86e

Please sign in to comment.