Manifest commit lock automation

[prudhvigodithi]

Description

Updates the manifest with commit ID's and creates a PR. There are 2 supported manifest lock actions

1. Match Build Manifest: Will update the input manifest with commit ID's from release candidate build manifest and raises a PR.
2. Update to Recent Commits: Will update the input manifest with component repo release branch head commit and raises a PR.

For both the above actions COMPONENTS option can be used to filter the desired action on the specific component.

Sample Test PR's

Update to RC commit ID'

https://github.com/prudhvigodithi/opensearch-build/pull/91/files

Update to Head Commit, filter by specific repos

https://github.com/prudhvigodithi/opensearch-build/pull/92/files

Issues Resolved

#4561

If you guys are good with the approach I will add unit tests and create the Jenkins job.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.39%. Comparing base (5dce365) to head (463b90f).
Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4583      +/-   ##
==========================================
+ Coverage   92.17%   92.39%   +0.22%     
==========================================
  Files         192      193       +1     
  Lines        6297     6314      +17     
==========================================
+ Hits         5804     5834      +30     
+ Misses        493      480      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[peterzhuamazon]

I feel like a lot of the code can be put into lib such as create pull request section being generic.

Also you can rebase on main to use the latest commits of link checker in #4581.

Thanks.

[prudhvigodithi]

I feel like a lot of the code can be put into lib such as create pull request section being generic.

Majority of the code here is only related to Manifest Lock Automation, not reusable to be part of the library, may be git create pull request we can and be re-used later in future.

        stage('Create Pull Request') {
            steps {
                script {
                    withCredentials([usernamePassword(credentialsId: 'jenkins-github-bot-token', passwordVariable: 'GITHUB_TOKEN', usernameVariable: 'GITHUB_USER')]) {
                        try {
                            sh """
                                git remote set-url origin https://prudhvigodithi:${GITHUB_TOKEN}@github.com/prudhvigodithi/opensearch-build
                                git config user.email "pgodithi@amazon.com"
                                git config user.name "prudhvigodithi"
                                git checkout -b manifest-lock
                        """
                            def status = sh(returnStdout: true, script: 'git status --porcelain')
                            if (status) {
                                sh """
                                    git add . && git commit -sm "Manifest Commit Lock"
                                    git push origin manifest-lock --force
                                    gh pr create --title '[${params.RELEASE_VERSION}] Manifest Commit Lock' --body 'Manifest Commit Lock for Release ${params.RELEASE_VERSION} ' -H manifest-lock -B main
                                """
                            } else {
                                println 'Nothing to commit!'
                            }
                        } catch (e) {
                            error 'An error occured while creating manifest commit lock' + e.toString()
                        }
                    }
                }
            }
        }

But if you are ok with idea of Manifest Lock Automation, I can go ahead and do the code optimization.

Thanks

[prudhvigodithi]

This Manual PR #4586 should be covered with this automation.

[prudhvigodithi]

I feel like a lot of the code can be put into lib such as create pull request section being generic.

Majority of the code here is only related to Manifest Lock Automation, not reusable to be part of the library, may be git create pull request we can and be re-used later in future.

        stage('Create Pull Request') {
            steps {
                script {
                    withCredentials([usernamePassword(credentialsId: 'jenkins-github-bot-token', passwordVariable: 'GITHUB_TOKEN', usernameVariable: 'GITHUB_USER')]) {
                        try {
                            sh """
                                git remote set-url origin https://prudhvigodithi:${GITHUB_TOKEN}@github.com/prudhvigodithi/opensearch-build
                                git config user.email "pgodithi@amazon.com"
                                git config user.name "prudhvigodithi"
                                git checkout -b manifest-lock
                        """
                            def status = sh(returnStdout: true, script: 'git status --porcelain')
                            if (status) {
                                sh """
                                    git add . && git commit -sm "Manifest Commit Lock"
                                    git push origin manifest-lock --force
                                    gh pr create --title '[${params.RELEASE_VERSION}] Manifest Commit Lock' --body 'Manifest Commit Lock for Release ${params.RELEASE_VERSION} ' -H manifest-lock -B main
                                """
                            } else {
                                println 'Nothing to commit!'
                            }
                        } catch (e) {
                            error 'An error occured while creating manifest commit lock' + e.toString()
                        }
                    }
                }
            }
        }

But if you are ok with idea of Manifest Lock Automation, I can go ahead and do the code optimization.

Thanks

Just realized converting this into library might not be as generic as I thought, because it will have multiple parameters like remote_repo_url , user_email , user_name, source_branch, destination_branch, pr_title, pr_body, force_push (true or false), adding more lines of code every time when imported and I can imagine more options like if/else and parameters that needs to be squeezed that are related to git operations. So implementing the right set of git actions as required directly is better than a library.

[prudhvigodithi]

@peterzhuamazon I think this is expected, failures with Link Checker OK (429 Too Many Requests): Too Many Requests.

[peterzhuamazon]

@peterzhuamazon I think this is expected, failures with Link Checker OK (429 Too Many Requests): Too Many Requests.

Just re-run after a few min, it will pass.

[gaiksaya]

Hi @prudhvigodithi
Can you add tests for either manifest commit update? Like only update opensearch manifest?
Also should we add tests for exact commit assertions? For example, 2.12.0 is released. So add test based to see if head commits match?

[prudhvigodithi]

Hi @prudhvigodithi Can you add tests for either manifest commit update? Like only update opensearch manifest? Also should we add tests for exact commit assertions? For example, 2.12.0 is released. So add test based to see if head commits match?

The Generated .txt file should have the commits, you want to re-assert them again?

[prudhvigodithi]

Hey @peterzhuamazon I have made your suggested changes and pushed in my latest commit, following are some test PR's with this automation behavior.

Manifest lock with RC Commit ID

https://github.com/prudhvigodithi/opensearch-build/pull/94/files

Manifest lock with latest HEAD Commit ID

https://github.com/prudhvigodithi/opensearch-build/pull/95/files

Manifest lock for specified individual components:

https://github.com/prudhvigodithi/opensearch-build/pull/96/files