Recording components logs and generate component yaml for smoke tests framework #5333
Conversation
Signed-off-by: Zelin Hao <zelinhao@amazon.com>
zelinh
requested review from
dblock,
peterzhuamazon,
gaiksaya,
rishabh6788,
prudhvigodithi,
Divyaasm and
tianleh
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5333 +/- ##
==========================================
+ Coverage 92.04% 92.07% +0.03%
==========================================
Files 202 202
Lines 7042 7069 +27
==========================================
+ Hits 6482 6509 +27
Misses 560 560 ☔ View full report in Codecov by Sentry. |
| @@ -64,6 +67,31 @@ def validate_response_swagger(self, response: Any) -> None: | |||
| validate_response(response=response, spec=self.spec_, request=request) | |||
| logging.info("Response is validated.") | |||
|
|
|||
| def record_test_result(self, component: str, test_api: str, api_action: str, stdout: str, stderr: str) -> None: | |||
| test_config = f"{api_action}_{test_api.replace('/', '_')}" | |||
There was a problem hiding this comment.
Can you explain what the actual line would be?
There was a problem hiding this comment.
In this case, test results folder will be looked like this:
-- test_results
-- smoke_tests
-- local_cluster_logs
-- stdout.txt
-- stderr.txt
-- security
-- GET___plugins__security_api_account
-- stdout.txt
-- stderr.txt
-- security.yml
-- PUT___plugins__security_api_actiongroups_my-action-group
-- stdout.txt
-- stderr.txt
-- security.yml
-- POST_...
-- ...
-- ...
There was a problem hiding this comment.
Do we really need stdout and stderr for each API? Feels like too many files for just a single API response.
There was a problem hiding this comment.
If the API test passes, the stderr will be empty and stdout will display all logging for this specific API path. There will be just two std files and a component yml file generated for each API.
There was a problem hiding this comment.
Each of the component yaml file(e.g. security.yml) will contain specific test results so I believe it's better to have them separately. WDYT @rishabh6788
| def setup_logging_buffers(self) -> Tuple[io.StringIO, io.StringIO, logging.StreamHandler, logging.StreamHandler]: | ||
| info_buffer = io.StringIO() | ||
| error_buffer = io.StringIO() | ||
|
|
||
| info_handler = logging.StreamHandler(info_buffer) | ||
| error_handler = logging.StreamHandler(error_buffer) | ||
|
|
||
| info_handler.setLevel(logging.INFO) | ||
| error_handler.setLevel(logging.ERROR) | ||
|
|
||
| logging.getLogger().addHandler(info_handler) | ||
| logging.getLogger().addHandler(error_handler) | ||
|
|
||
| return info_buffer, error_buffer, info_handler, error_handler |
There was a problem hiding this comment.
What values are re getting from this?
There was a problem hiding this comment.
This method is to use log buffer and handler to record logging information in order to generate a output file.
| @@ -64,6 +67,31 @@ def validate_response_swagger(self, response: Any) -> None: | |||
| validate_response(response=response, spec=self.spec_, request=request) | |||
| logging.info("Response is validated.") | |||
|
|
|||
| def record_test_result(self, component: str, test_api: str, api_action: str, stdout: str, stderr: str) -> None: | |||
| test_config = f"{api_action}_{test_api.replace('/', '_')}" | |||
There was a problem hiding this comment.
Do we really need stdout and stderr for each API? Feels like too many files for just a single API response.
Description
Recording components logs and generate component yaml for smoke tests framework
Issues Resolved
Part of #5320
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.