Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Enable test record and results report for smoke test framework #5320

Open
3 tasks
zelinh opened this issue Feb 13, 2025 · 1 comment
Open
3 tasks
Assignees
Labels
enhancement New Enhancement test

Comments

@zelinh
Copy link
Member

zelinh commented Feb 13, 2025

Overview

With the Smoke Test Framework (issue #5223) now established, we have successfully onboarded multiple OpenSearch plugins—including Index Management, Security, SQL, and K-NN—to ensure reliable pre-release validation.

To further improve integration and visibility, we aim to enhance the smoke tests with a reporting and notification mechanism. Ideally, after each run, the framework should generate a structured test report similar to the integration test report, capturing test results and associated logs.

The generated report will be:

  • Improving visibility into test results.
  • Consumed by metric dashboards for more insights.
  • Used for automated failure notifications, enabling faster issue resolution.

Tasks

  • Enhance the smoke test workflow to record test results and generate a structured component YAML file.
  • Integrate with the reporting workflow to produce an indexable test-report.yml containing test outcomes.
  • Work on indexing the new test report into existing metric dashboards for monitoring and analysis.

This improvement will provide greater transparency, automation, and efficiency in OpenSearch distribution testing.

@zelinh zelinh added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Feb 13, 2025
@zelinh zelinh self-assigned this Feb 13, 2025
@zelinh zelinh added test and removed untriaged Issues that have not yet been triaged labels Feb 13, 2025
@zelinh zelinh moved this from 🆕 New to 🏗 In progress in Engineering Effectiveness Board Feb 14, 2025
@zelinh
Copy link
Member Author

zelinh commented Feb 17, 2025

I'm planning to record the component smoke test results using the following hierarchy for test_results:

-- test_results
  -- smoke_tests
       -- local_cluster_logs
            -- stdout.txt
            -- stderr.txt
       -- security 
            -- GET_plugins_security_api_account
                -- stdout.txt
                -- stderr.txt
                -- security.yml
            -- PUT_plugins_security_api_actiongroups_my-action-group
                -- stdout.txt
                -- stderr.txt
                -- security.yml
            -- POST_...
       -- ...
       -- ...

This structure ensures that logs and test results are well-organized by component and API, making it easier to track failures and debug issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New Enhancement test
Projects
Status: 🏗 In progress
Development

No branches or pull requests

1 participant