Add Deployment workflows #519

Divyaasm · 2025-01-28T23:00:52Z

Description

Adds changeset and deployment workflows to deploy infra stacks changes with every Pull request

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Divya Madala <divyaasm@amazon.com>

peterzhuamazon · 2025-01-28T23:54:27Z

.github/workflows/prod-deploy.yml

+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, 'Approve Prod Deployment')


Can anyone write this line and approve prod?

Yes Peter, we can control who can approve just by adding their gh alias in the above condition. For example github.actor == 'Divyaasm'.

peterzhuamazon · 2025-01-28T23:54:42Z

.github/workflows/prod-deploy.yml

+    if: >-
+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, 'Approve Prod Deployment')
+    name: Get approval for deployment


Name should be beta deployment workflow?

Sure, Will Change to avoid confusion.

peterzhuamazon · 2025-01-28T23:54:54Z

.github/workflows/beta-deploy.yml

+    if: >-
+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, 'Approve Beta Deployment')
+    name: Get approval for deployment


Same as above.

gaiksaya

It would be helpful to see this is action. Maybe you can share something you tried in your fork?

I highly recommend using GitHub environments for deployments https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment
This would also help to keep secrets to different env separate from each other so that blast radius is minimum. Each env can also have approvers that need to approve the workflow before proceeding.

The prod deployment workflow can be an after action and only runs when beta is succeeded.

Another approach to consider: For beta as well as prod deployment, we can get approvals using something like opensearch-project/opensearch-java#1354 with change-set added in the issue body showing what all changes are gonna happen.

gaiksaya · 2025-01-29T00:46:20Z

.github/workflows/changeset.yml

+      - name: Create Change Set for Beta
+        run: |
+          npm run cdk diff -- OpenSearch-CI-Config-Beta -c useSsl=true -c serverAccessType=prefixList -c restrictServerAccessTo=${{ secrets.PREFIX_LIST }}
+          npm run cdk diff -- OpenSearch-CI-Beta -c useSsl=true -c authType=github -c dataRetention=true -c macAgent=true -c useProdAgents=true -c enableViews=true -c ignoreResourcesFailures=false -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}}
+
+      - name: Assume IAM Role for Prod
+        uses: aws-actions/configure-aws-credentials@v4.0.2
+        with:
+          role-to-assume: ${{ secrets.PROD_DEPLOYMENT_ROLE }}
+          aws-region: us-east-1
+
+      - name: Create Change Set for Prod
+        run: |
+          npm run cdk diff -- OpenSearch-CI-Config-Prod -c useSsl=true -c serverAccessType=prefixList -c restrictServerAccessTo=${{ secrets.INTERNET_ACCESS }}
+          npm run cdk diff -- OpenSearch-CI-Prod -c useSsl=true -c authType=github -c dataRetention=true -c macAgent=true -c useProdAgents=true -c enableViews=true -c ignoreResourcesFailures=false -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}}


What is the point of change sets here?
Just to see the diff? IF yes, maybe we should comment on the PR causing the workflow to run?

Yes just to see the changesets involved. It's triggered when a PR is raised. Should it be only trigerred if required through comment action?

gaiksaya · 2025-01-29T00:46:54Z

.github/workflows/beta-deploy.yml

+on:
+  issue_comment:
+    types: [created]


We want to deploy on merge from main or dispatch action. Can you please elaborate on this trigger?

The point of approval workflow is to add a manual step between PR merge and Deployment of the changes from the PR to Beta followed by Prod.
The changes in the current PR will

Trigger a changeset workflow to review the resources that changed/added/deleted in both beta and prod environments.

After the PR is merged, folks from the team would again confirm the changeset and comment "Approve ...." sequentially to proceed beta and prod deployment.

gaiksaya · 2025-01-29T00:49:11Z

.github/workflows/beta-deploy.yml

+          COMMAND_OUTPUT=$(npm run cdk diff -- OpenSearch-CI-Config-Beta -c useSsl=true -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}})
+          echo "$COMMAND_OUTPUT" | grep -q "Number of stacks with differences: 0"
+          if [ $? -eq 0 ]; then
+            echo "No stack changes involved"
+          else
+            npm run cdk deploy -- OpenSearch-CI-Config-Beta -c useSsl=true -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}}
+            npm run cdk deploy -- OpenSearch-CI-Beta -c useSsl=true -c authType=github -c dataRetention=true -c macAgent=true -c useProdAgents=true -c enableViews=true -c ignoreResourcesFailures=false -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}}


Can we create a deployment folder in this repo that would have cdk.context.json for these values?
I believe we would have more values coming up. For easy readability and maintenance as well as review we having a context.json file should help for each env. Both BETA and PROD.

Yeah Sayali, that's on my to-do list. Can we take this as further enhancement?

Divyaasm · 2025-01-30T01:28:49Z

It would be helpful to see this is action. Maybe you can share something you tried in your fork?

I highly recommend using GitHub environments for deployments https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment This would also help to keep secrets to different env separate from each other so that blast radius is minimum. Each env can also have approvers that need to approve the workflow before proceeding.

The prod deployment workflow can be an after action and only runs when beta is succeeded.

Another approach to consider: For beta as well as prod deployment, we can get approvals using something like opensearch-project/opensearch-java#1354 with change-set added in the issue body showing what all changes are gonna happen.

You can check sample action runs here Sayali

Signed-off-by: Divya Madala <divyaasm@amazon.com>

gaiksaya · 2025-02-11T23:13:42Z

.github/workflows/changeset.yml

+          echo "## CI-Config Stack Changeset" >> $GITHUB_OUTPUT
+          npm run cdk diff -- OpenSearch-CI-Config-${{ matrix.environment }} -c useSsl=true -c serverAccessType=prefixList -c restrictServerAccessTo=${{ secrets.PREFIX_LIST }} | sed -E 's/[0-9]{12}/[MASKED]/g' >> $GITHUB_OUTPUT
+
+          echo "" >> $GITHUB_OUTPUT
+          echo "## CI Stack ChangeSet ${{ matrix.environment }}" >> $GITHUB_OUTPUT
+          npm run cdk diff -- OpenSearch-CI-${{ matrix.environment }} -c useSsl=true -c authType=github -c dataRetention=true -c macAgent=true -c useProdAgents=true -c enableViews=true -c ignoreResourcesFailures=false -c serverAccessType=prefixList -c restrictServerAccessTo=${{secrets.PREFIX_LIST}} | sed -E 's/[0-9]{12}/[MASKED]/g' >> $GITHUB_OUTPUT


Lets not use context variables. Maybe create a new PR with properties set as desired?
Lets get that PR in first which will simply the deployment and diff commands significantly.

gaiksaya · 2025-02-11T23:14:00Z

.github/workflows/changeset.yml

+          role-to-assume: ${{ secrets.CHANGESET_ROLE }}
+          aws-region: us-east-1
+
+      - name: Create Change Set for Beta


Suggested change

- name: Create Change Set for Beta

- name: Create Change Set for ${{matrix.environment}}

Add Deployment workflows

998c04b

Signed-off-by: Divya Madala <divyaasm@amazon.com>

Divyaasm requested review from peterzhuamazon, gaiksaya, rishabh6788, zelinh and prudhvigodithi as code owners January 28, 2025 23:00

peterzhuamazon reviewed Jan 28, 2025

View reviewed changes

gaiksaya requested changes Jan 29, 2025

View reviewed changes

Divyaasm had a problem deploying to Prod February 11, 2025 19:10 — with GitHub Actions Failure

Divyaasm had a problem deploying to Beta February 11, 2025 19:10 — with GitHub Actions Failure

Update deployment workflows

5704841

Signed-off-by: Divya Madala <divyaasm@amazon.com>

Divyaasm force-pushed the workflow branch from be330e6 to 5704841 Compare February 11, 2025 19:10

Divyaasm had a problem deploying to Beta February 11, 2025 19:11 — with GitHub Actions Failure

Divyaasm had a problem deploying to Prod February 11, 2025 19:11 — with GitHub Actions Failure

gaiksaya reviewed Feb 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Deployment workflows #519

Add Deployment workflows #519

Divyaasm commented Jan 28, 2025

peterzhuamazon Jan 28, 2025

Divyaasm Jan 30, 2025

peterzhuamazon Jan 28, 2025

Divyaasm Jan 30, 2025

peterzhuamazon Jan 28, 2025

gaiksaya left a comment

gaiksaya Jan 29, 2025

Divyaasm Jan 30, 2025

gaiksaya Jan 29, 2025

Divyaasm Jan 30, 2025 •

edited

Loading

gaiksaya Jan 29, 2025

Divyaasm Jan 30, 2025

Divyaasm commented Jan 30, 2025

gaiksaya Feb 11, 2025

gaiksaya Feb 11, 2025 •

edited

Loading

		github.event.issue.pull_request &&
		contains(github.event.comment.body, 'Approve Prod Deployment')

	- name: Create Change Set for Beta
	- name: Create Change Set for ${{matrix.environment}}

Add Deployment workflows #519

Are you sure you want to change the base?

Add Deployment workflows #519

Conversation

Divyaasm commented Jan 28, 2025

Description

Issues Resolved

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaiksaya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Divyaasm Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Divyaasm commented Jan 30, 2025

Choose a reason for hiding this comment

gaiksaya Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

Divyaasm Jan 30, 2025 •

edited

Loading

gaiksaya Feb 11, 2025 •

edited

Loading