Onboard all OpenSearch plugins to Smoke tests framework

[zelinh]

Background

With the smoke test framework (#5223) now established, we have successfully onboarded multiple OpenSearch plugins—including Index Management, Security, SQL, and K-NN—to the framework (PR #5255).

To maximize the effectiveness of smoke testing, we aim to onboard all OpenSearch plugins into the framework. This effort will enhance overall testing coverage, ensuring greater reliability and functionality validation for OpenSearch and its plugins.

Overview of Smoke Tests

Smoke tests verify the basic functionality of an OpenSearch distribution bundle with all plugins installed. Each plugin has its own test specification YAML file, stored in the OpenSearch Build repository and organized by major version.

These YAML files list API test cases, which the smoke test workflow executes. The results are then validated against the OpenSearch API Specification to ensure API correctness and expected behavior.

How to Onboard a Plugin

To onboard your plugin to the smoke test framework, follow the Onboarding Guide.

Steps to Onboard:

1. Add a YAML test file

   • Create a new test file for your plugin in the relevant major version directory under smoke_tests_spec.
   • Example: security.yml.

2. Define API tests in the YAML file

   • Specify:
     • API path
     • HTTP method
     • Request body (if applicable)
     • Custom Content-Type if different from application/json.
   • See an example test definition here.

3. Ensure the APIs are onboarded to the OpenSearch API Specification repository

   • Before adding APIs to smoke tests, they must be present in the OpenSearch API Specification repository.
   • Refer to the Developer Guide for onboarding steps.

4. Reference the test file in the test manifest

   • Ensure the newly added YAML file is referenced in the test manifest for your plugin’s component.

Collaboration with Plugin Owners:

We are reaching out to all OpenSearch plugin owners to facilitate onboarding. We will provide:
✅ Documentation and guidelines for onboarding.
✅ Support in setting up test cases.
✅ Assistance in troubleshooting issues.

Expected Outcome:

• Stronger OpenSearch releases with better validation of plugins.
• Standardized, reusable testing framework for ongoing plugin validation.
• Faster detection of integration issues before release.

Plugin Onboarding Status

We will track the onboarding progress of each plugin here:

Component Name 📛	Onboarding status ❓	Notes 📓
OpenSearch	✅	#5255
common-utils	❌	opensearch-project/common-utils#791
opensearch-learning-to-rank-base	❌	opensearch-project/opensearch-learning-to-rank-base#141
opensearch-remote-metadata-sdk	❌	opensearch-project/opensearch-remote-metadata-sdk#85
ml-commons	❌	opensearch-project/ml-commons#3541
job-scheduler	❌	opensearch-project/job-scheduler#731
opensearch-reports	❌	opensearch-project/reporting#1070
security	✅	#5255
performance-analyzer	❌	opensearch-project/performance-analyzer#787
opensearch-observability	❌	opensearch-project/observability#1902
k-NN	✅	#5255
neural-search	❌	opensearch-project/neural-search#1186
geospatial	❌	opensearch-project/geospatial#724
anomaly-detection	❌	opensearch-project/anomaly-detection#1420
cross-cluster-replication	❌	opensearch-project/cross-cluster-replication#1505
notifications	❌	opensearch-project/notifications#1000
alerting	❌	opensearch-project/alerting#1796
sql	✅	#5255
asynchronous-search	❌	opensearch-project/asynchronous-search#704
security-analytics	❌	opensearch-project/security-analytics#1474
index-management	✅	#5255
flow-framework	❌	opensearch-project/flow-framework#1063
opensearch-system-templates	❌	opensearch-project/opensearch-system-templates#58
custom-codecs	❌	opensearch-project/custom-codecs#225
skills	❌	opensearch-project/skills#526
user-behavior-insights	❌	opensearch-project/user-behavior-insights#77
query-insights	❌	opensearch-project/query-insights#244

📢 Plugin owners: We encourage you to onboard your plugin to the smoke test framework! If you have any questions or need assistance, feel free to comment on this issue or reach out.

Let’s work together to strengthen OpenSearch’s testing and release process! 🚀