[Bug]: OSD Integration Tests: Workflow Check and OpenSearch Validation

[prudhvigodithi]

Describe the bug

The integ test workflow for OpenSearch Dashboards requires OpenSearch to be full up and running before running tests through OpenSearch Dashboards. Without OpenSearch cluster validation the integ tests are sometimes flaky and fail on that specific OSD component

Sample Error

2023-07-12 23:20:46 INFO     Pinging http://localhost:5601/api/status

2023-07-12 23:20:46 INFO     Service not available, yet

2023-07-12 23:20:56 INFO     Sending SIGKILL to PID 219

2023-07-12 23:20:56 INFO     Process killed with exit code 1

2023-07-12 23:20:56 INFO     Sending SIGKILL to PID 240

2023-07-12 23:20:56 INFO     Process killed with exit code None

Error log: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.9.0/6382/linux/x64/tar/test-results/3680/integ-test/customImportMapDashboards/with-security/stdout.txt

This was shown up for customImportMapDashboards during the 2.9.0 release where customImportMapDashboards failed and upon re-run passed the next time.

To reproduce

This is flaky but should show when running the integ test workflow for OSD components.

Sample Failed build:
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/3682/pipeline/

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[peterzhuamazon]

[Triage] This needs some deep because the OSD without security plugin is able to connect to OS cluster, which mean the OS cluster has also disabled security plugin during without-security test.

We need some logs from the failure which we can pinpoint which part of the cluster deployment is not properly setup, in order to raise 401 issues during OSD integTest for multiple plugins.

[Divyaasm]

The Issue could be solved once the Build workflow is integrated with Validation Workflow. And the Integration tests are run only for the builds that pass the Validation which usually includes spinning up the OS and OSD cluster.
#3441

[Divyaasm]

Resolved by #4424 . The given artifacts are now validated first, if the cluster successfully comes up for OS and OSD Integ tests are triggered.