tests reorganization and add main test script (#307)

* tests reorganization and add main test script

1 - reorganization of the tests folder
    tests having same category live in a subfolder

2 - expose pytest capabilities to users
    currently, tox is masking all pytest features and only
    expose what we specify in tox.ini file
    since tox is only here to manage the test environment
    it should be transparent to the users for tests execution
    and management. the users should be able to take full advantage
    of what pytest offers.
    we introduced a new script tdtest that hides tox and exopose
    all pytest features to users

* Update tests/README.md

Co-authored-by: Will French <will.french@canonical.com>

* update tdtest param

---------

Co-authored-by: Will French <will.french@canonical.com>

tests/README.md

@@ -4,88 +4,30 @@ This folder contains Intel TDX tests.
 
 ### Pre-requisites
 
-- The tests must be executed a host that has been setup properly for Intel TDX.
+- The tests must be executed on a host that has been set up properly for Intel TDX.
 
-- Tox must be installed along with python3:
-```
-$ sudo apt install tox
-$ sudo apt install python3
-```
-
-- You must specify a path to the guest image with `TDXTEST_GUEST_IMG` environment variable.
-  This is for both pytest and checkbox tests.
-
-- The guest image must enable ssh server with password-based authentication for `root` user.
-  The root user password must be `123456`
-
-### Run tests with tox/pytest
-
-Go to the `tests` folder.
+### Run tests
 
-- Run sanity tests to check the host setup:
+The script `tdtest` is the test runner, it is a wrapper around `pytest`.
+Please run `tdtest -h` for more details about its usage.
 
-```
-$ sudo -E tox -e test_host
-```
+The tests are organized in different categories and this organization is
+reflected in the structure of the `tests`.
 
-- Run guest tests:
+You can choose to run a category of tests by specifying the appropriate sudirectory under `tests`. For example, to run the boot tests:
 
 ```
-$ sudo -E tox -e test_guest
+$ sudo ./tdtest tests/boot
 ```
 
-- Run boot tests:
+You can run all tests except the performance:
 
 ```
-$ sudo -E tox -e test_boot
+$ sudo ./tdtest -k 'not test_perf'
 ```
 
-- Run perf tests:
-
-```
-$ sudo -E tox -e test_perf
-```
-
-- Run quote tests:
-
-```
-$ sudo -E tox -e test_quote
-```
-
-- Run stress tests:
-
-```
-$ sudo -E tox -e test_stress
-```
-
-- Run all tests:
-
-Please note that the performance tests can take a long time (order of magnitude of a few hours per `pytest.test_perf_benchmark`) to run.
-
-```
-$ sudo -E tox -e test_all
-```
-
-To list tests without running:
-
-```
-sudo -E tox -e collect_tests -- EXPRESSION
-```
-
-`EXPRESSION` is the expression provided to the argument `-k` of `pytest`.
-For more details on its format, you can refer to `pytest` help page.
-
-For example:
-
-```
-sudo -E tox -e collect_tests -- 'test_guest or test_stress'
-```
-
-To run tests matching an expression:
-
-```
-sudo -E tox -e test_specify -- EXPRESSION
-```
+Since `tdtest` is a wrapper of pytest, it exposes all the features of `pytest`
+that you can use to run, manage and inspect the tests.
 
 ### Run tests with checkbox:
 

tests/tdtest

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+usage() {
+    cat <<EOM
+This script executes tests for TDX
+
+It is a wrapper around pytest, pytest being the framework used to run the tests.
+It makes use of tox to manage the test execution environment.
+It also provides additional arguments on top of pytest arguments.
+
+So in addition to its specific arguments listed below, the script will
+pass through all the arguments to pytest (please check pytest -h to see
+available argument).
+
+NB: all pytest arguments must come together AFTER the arguments of tdtest.
+
+Usage: $(basename "$0") [-i|-h] pytest_args
+  -i|--td-image <image>     Ubuntu guest image
+  -h|--help                 Show this help
+
+Examples:
+  To run tests in boot folder:
+    $ sudo $(basename "$0") tests/boot
+  To list tests in boot folder:
+    $ sudo $(basename "$0") --co tests/boot
+  To run all the tests:
+    $ sudo $(basename "$0")
+
+---
+EOM
+}
+
+parse_params() {
+    while :; do
+        case "${1-}" in
+        -h | --help)
+            usage
+            exit 0
+            ;;
+        -i | --td-image)
+            TDXTEST_GUEST_IMG=$(realpath "${2-}")
+            shift
+            ;;
+        "")
+            break
+            ;;
+        *)
+            # other arguments will be passed to pytest
+            pytest_args="$@"
+            break
+            ;;
+        esac
+        shift
+    done
+}
+
+parse_params "$@"
+
+if [ "$EUID" -ne 0 ]
+  then echo "Please run as root"
+  exit
+fi
+
+if [[ -z "${UBUNTU_VER}" ]]; then
+    UBUNTU_VER=$(lsb_release -rs)
+fi
+
+if [[ ! -f "${TDXTEST_GUEST_IMG}" ]]; then
+    TDXTEST_GUEST_IMG=$PWD/../guest-tools/image/tdx-guest-ubuntu-${UBUNTU_VER}-generic.qcow2    
+fi
+
+apt install -y python3 tox
+
+export TDXTEST_GUEST_IMG
+
+echo "Run tests with TD image: ${TDXTEST_GUEST_IMG}"
+echo "  pytest args: ${pytest_args}"
+tox -- --ignore=lib/ "${pytest_args}"

tests/tests/test_guest_report.py → tests/tests/report/test_guest_report.py

@@ -34,6 +34,6 @@ def test_guest_report(qm):
 
     deploy_and_setup(m)
 
-    m.check_exec(f'python3 {guest_workdir}/tests/guest/test_tdreport.py')
+    m.check_exec(f'python3 {guest_workdir}/lib/guest/test_tdreport.py')
 
     qm.stop()

tests/tox.ini

@@ -1,6 +1,6 @@
 [tox]
 envlist = 
-  test_host
+  tdx
 isolated_build = True
 skip_missing_interpreters = True
 
@@ -22,42 +22,6 @@ commands_pre =
   bash -c 'if [ `id -u` -ne 0 ]; then echo "Must run all tests with sudo" 1>&2; (exit -1); fi'
   {toxinidir}/tox/setup-env-tox.sh
 
-[testenv:test_guest]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_guest_report.xml --ignore=tests/guest -k test_guest
-
-[testenv:test_host]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_host_report.xml --ignore=tests/guest -k test_host
-
-[testenv:test_boot]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_boot_report.xml --ignore=tests/guest -k test_boot
-
-[testenv:test_perf]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_perf_report.xml --ignore=tests/guest -k test_perf
-
-[testenv:test_quote]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_quote_report.xml --ignore=tests/guest -k test_quote
-
-[testenv:test_stress]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_stress_report.xml --ignore=tests/guest -k test_stress
-
-[testenv:test_all_except_perf]
-commands =
-  python3 -m pytest -s -v --junitxml=test_all_report.xml --ignore=tests/guest -k 'not test_perf'
-
-[testenv:test_all]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_all_report.xml --ignore=tests/guest
-
-[testenv:test_specify]
-commands = 
-  python3 -m pytest -s -v --junitxml=test_specify_report.xml --ignore=tests/guest -k {posargs}
-
-[testenv:collect_tests]
+[testenv:tdx]
 commands =
-  python3 -m pytest -s -v --collect-only --ignore=tests/guest -k {posargs}
+  python3 -m pytest -s -v {posargs}