[Feature] Add ability to run test scenarios in the Jenkins runner environment

[cwperks]

Is your feature request related to a problem? Please describe

I would like to have the ability to run a single test, test suite or gradle test task in the Jenkins runner multiple times to help reproduce and minimize flaky tests. When looking at various flaky test issues, I run the test locally multiple times and the tests pass every time making it harder to simulate the conditions that led to failure on a CI check.

Describe the solution you'd like

When looking at flaky test issues, it could be helpful to run the tests on the Jenkins runner (or any Github runner) to simulate similar conditions that led to failure. I'd like the ability to not run the full gradle check, but rather a single test, test suite or gradle test task (like :server:test) without running the full gradle check.

Related component

Build

Describe alternatives you've considered

Continue to try to reproduce locally or on remote instances of different os types.

Additional context

The security plugin added a RepeatRule to help with local testing for diagnosing flaky tests. It would be helpful to have something similar and have it run on the Jenkins instances or a Github Runner.

[peternied]

@opensearch-project/admin Can you transfer this to opensearch-build? This seems like a feature of how the jenkins job can be run or scheduled

[prudhvigodithi]

[Triage]
Thanks @cwperks, to have an ability to run a job in Jenkins requires a job execution access for a user and the best way to do it is to have a mechanism for Jenkins to be able to login with GitHub and the restricted GitHub users can be part of the job execution role, but even with this it would be limited to only restricted users. The entire build/test jobs are all reproducible locally (with or without the same docker images used in Jenkins) @cwperks may I please know what are the challenges when ran locally? is it the compute limits ?
Also ya its a good idea to run the targeted test in gradle check as required (even during the PR creation), but at this point since the only test for Core is the gradle check its not targeted at the test level rather a full blown gradle check is executed for every PR and post merge actions. I'm happy to see and contribute to reduce the flakiness and time takes to run the gradle check.

Thank you
@gaiksaya @rishabh6788 @peterzhuamazon @bbarani

[bbarani]

@cwperks As mentioned by @prudhvigodithi you can setup your own Jenkins instance by following the steps mentioned here.