From 2c67bbe397a2a820def3729922d539e5cee2143a Mon Sep 17 00:00:00 2001 From: AlexandraTrifan Date: Mon, 3 Feb 2025 15:42:04 +0200 Subject: [PATCH] docs/tests: add a testing guidelines document with detailed steps for the testing process. Signed-off-by: AlexandraTrifan --- docs/tests/index.rst | 1 + docs/tests/plugins/index.rst | 3 - docs/tests/testing_guide.rst | 132 +++++++++++++++++++++++++++++++++++ testing_results/README.md | 1 + tools/exclude_list.sh | 1 + 5 files changed, 135 insertions(+), 3 deletions(-) create mode 100644 docs/tests/testing_guide.rst create mode 100644 testing_results/README.md diff --git a/docs/tests/index.rst b/docs/tests/index.rst index 71b0951ead..a3e6c44083 100644 --- a/docs/tests/index.rst +++ b/docs/tests/index.rst @@ -10,5 +10,6 @@ Contents :includehidden: :maxdepth: 3 + testing_guide plugins/index general/index diff --git a/docs/tests/plugins/index.rst b/docs/tests/plugins/index.rst index fef3f1f0cd..555b8efbb3 100644 --- a/docs/tests/plugins/index.rst +++ b/docs/tests/plugins/index.rst @@ -3,9 +3,6 @@ Plugins Test Cases =============================================================================== -* :ref:`Datalogger ` -* :ref:`Register Map ` - Contents --------------------------------------------------------------------- diff --git a/docs/tests/testing_guide.rst b/docs/tests/testing_guide.rst new file mode 100644 index 0000000000..257cd22e21 --- /dev/null +++ b/docs/tests/testing_guide.rst @@ -0,0 +1,132 @@ +.. _testing_guide: + +Testing Guide +=============================================================================== + +Test cases, test results and test reports for Scopy are found in the sorce code +repository. This approach allows for the test cases to be versioned and +maintained along with the source code. + +Overview +--------- + +The test cases are written in reStructuredText format and are +located in the `scopy/docs/tests` directory. The test results and reports are +generated by the test runner and are located in the `scopy/testing_results` directory. + +The test cases are organized in the following structure: + - `docs/tests/index.rst` - the main index file for the test cases. + - `docs/tests/testing_guide.rst` - this file. + - `docs/tests/plugins/index.rst` - the index file for the plugin test cases. + - `docs/tests/general/index.rst` - the index file for the general test cases. + - `docs/tests/plugins//index.rst` - the index file for the plugin test cases. + - `docs/tests/plugins//.rst` - the test suite file for the plugin. + + +A test suite is a collection of tests that are related to a specific plugin or instrument. +Each test suite has a setup environment section that describes the steps to setup the environment +for the tests in the suite. Multiple scenarios can be described in the setup environment section. + +At the beginning of each test suite, a note is included that specifies the user guide for the +plugin. The steps do not include detailed information about each feature, it is expected that the +tester is familiar with the above mentioned user guide. + +Below the note, a list table is included that specifies the tester, test date, Scopy version, +plugin version and comments. This table is completed by the tester during the testing phase and +it is used for test report generation. + +Each test case has a unique identifier (UID) that is used to reference the test case in the test results +and reports. The test case description is a brief description of the test case. +The test case preconditions are the requirements that need to be met before running the test case. +This section includes a link specifying the setup environment for the test case. + +The steps section describes the steps to be taken to run the test case. Multiple steps can +have an expected result and an actual result. The expected result is the expected outcome of the +step. The actual result is the actual outcome of the step and will be completed during testing, +by the tester, as described in this document. + +The tested OS section specifies the operating system on which the test was run and it must be +completed according to the OS information provided in the Preconditions section. If the test +is specifically written for a certain OS, the OS information will be provided in the Preconditions +section and the tested OS needs to be the same. Other options can be: "ANY" (the outcome is the +same on all OSes, so testing one of them is sufficient) or "ALL" (it is necessary to run the test +separately on each OS). This section is also completed during testing. + +The comments and the result sections are completed by the tester during testing. The comments +section can include any additional information about the test case. The result section specifies +whether all expected results were met during testing. The result can be "PASS" or "FAIL". + + +Testing process in detail +--------------------------- + +Example: we are testing Scopy v3.0.0. + +The testing process is as follows: + +1. Navigate to the `official Scopy repository `__. +2. From the **main** branch create a new branch named **testing_v3.0.0**. + +.. image:: https://raw.githubusercontent.com/analogdevicesinc/scopy/doc_resources/resources/testing/create_new_branch.png + :align: center +.. + +3. On the newly created branch, make a copy of the `docs/tests` directory + in the `testing_results` directory. Rename the new folder to `testing_results_v3.0.0`. +4. In the `testing_results_v3.0.0` directory, choose the test case that needs to be run. + For this example we will consider the ADC tests, located in plugins/adc/adc_tests.rst. +5. Open the `adc_tests.rst` file. +6. Enter the Github Edit mode (edit in place) to be able to edit the file. + +.. image:: https://raw.githubusercontent.com/analogdevicesinc/scopy/doc_resources/resources/testing/github_edit_mode.png + :align: center +.. + +7. First thing that needs completion is the table containing information on the tester, + test date, Scopy version, plugin version and comments. + This table is located at the beginning of the file. + Check the screenshot below on where to modify data inside the table and what's the + outcome after committing. + +.. image:: https://raw.githubusercontent.com/analogdevicesinc/scopy/doc_resources/resources/testing/update_table_date.png + :align: center +.. + +8. After making some changes to a file, you can commit them and resume the testing later. + In order to commit the changes, click the top right **Commit changes...** button. + A popup will appear requesting a commit title and an optional description. + Make sure to **check the "Commit directly to the testing_v3.0.0 branch"** option + before clicking the **Commit changes** button. + +.. image:: https://raw.githubusercontent.com/analogdevicesinc/scopy/doc_resources/resources/testing/committing_changes.png + :align: center +.. + +9. The last step will create a new commit on the testing branch. There can be multiple + commits before the testing is finished. This is actually recommended as multiple + persons can test multiple instruments in parallel. + +10. While going through each test, complete the missing information accordingly. + The missing information is the actual result for each step and the tested OS. + The actual result is the outcome of the step and the tested OS is the operating + system on which the test was run. + The Comment section can be left empty if there is no additional information to add. + The Result section will be modified to either show **PASS** or **FAIL** according + to the actual results from the steps reproduced during testing. + +.. image:: https://raw.githubusercontent.com/analogdevicesinc/scopy/doc_resources/resources/testing/actual_result_tests.png + :align: center +.. + + As seen in the example above, the first step failed, so the result is FAIL + even if the second one passed. In the Comments section, it is mentioned what + might be the cause of the failure. + +11. Commit all the changes often to reduce the risk of losing any data. + +12. After all the tests are completed, a **Pull Request (PR)** will be created, having + having as target the **main** branch. The PR description will contain any + relevant information about testing. + The PR will be reviewed by the Scopy team and merged into the **main** branch. + The failures will be addressed, bugs will be fixed and the testing process will + be repeated until the final release candidate. \ No newline at end of file diff --git a/testing_results/README.md b/testing_results/README.md new file mode 100644 index 0000000000..86f78394c2 --- /dev/null +++ b/testing_results/README.md @@ -0,0 +1 @@ +Check the testing process guidelines on our [official documentation](https://analogdevicesinc.github.io/scopy/tests/testing_guide.html) page. diff --git a/tools/exclude_list.sh b/tools/exclude_list.sh index 9a73ccfcbb..0ae2218d63 100644 --- a/tools/exclude_list.sh +++ b/tools/exclude_list.sh @@ -163,5 +163,6 @@ GITINORE_FILES_LIST=( # plugins/regmap/.gitinore plugins/regmap/include/regmap/scopy-regmap_export.h plugins/regmap/include/regmap/scopy-regmap_config.h + testing_results/* ) GITINORE_FILES_LIST=("${GITINORE_FILES_LIST[@]/#/*\/}")