tests: new organization for nested tests (canonical#9098)

* New work ir organization for nested execution

* Fixes after merge

* Remove image dir call

* Fixes for tests

* Each image has an id to be distint for other images created manually

* Fix shell check error

* Save and restore compressed images

* Fixes for tests which failed

* Just keep 1 image

* Small fixes for default run

* Add checks for channels

* Fix issues found during execution

* Add new check for hotplug

* As core could be installed from any source, the version could be anything

It is tested on listing test so here just verify the channel is correct

* Fix core version check for core built from current branch

* test fix

* Fix shellcheck issue in spread.yaml

* New documentation for nested tests

* Update default value for SPREAD_BUILD_SNAPD_FROM_CURRENT var

* Fix documentation

* Using LOGS_DIR to read serial.log in spread.yaml

* Using local vars on functions and prefix NESTED_ for global vars

* Fix global variable without NESTED_ prefix

* Fix another var without NESTED_ prefix

* break case in case the system is rebooted after refresh or revert

* Update var names in github actions configuration

* Update vars without NESTED_

* Add another NESTED_ prefix

* Fix for cloud-init test

* Updating documentation with new names for variables

* Updating the name of the nested functions with prefix nested_

* Making simpler the management of the images and the compression

* Update variable names, moving SPREAD_NESTED_ to NESTED_

* Gracefull shutdown for the vm

* Fix shellcheck

* fix missing import

* Support parameters on generic function nested_get_image_url_for_vm

* Shell check fix

* Fix shellcheck of function

* Sync after the users are created to avoid issue after restart the vm

* Minor fix to locate correctly the assets

* Updating function names

* Fix for {} on function nested_configure_cloud_init_on_core20_vm

* Updating nested_execute function name

* Small fixes addressing review comments

* Adding 3 small fixes

.github/workflows/test.yaml

@@ -279,10 +279,10 @@ jobs:
       run: |
           # Register a problem matcher to highlight spread failures
           echo "::add-matcher::.github/spread-problem-matcher.json"
-          export SPREAD_BUILD_SNAPD_FROM_CURRENT=true
-          export SPREAD_ENABLE_KVM=true
+          export NESTED_BUILD_SNAPD_FROM_CURRENT=true
+          export NESTED_ENABLE_KVM=true
           if [ "${{ matrix.system }}" = "ubuntu-20.04-64" ]; then
-              export SPREAD_ENABLE_KVM=false
+              export NESTED_ENABLE_KVM=false
           fi
           spread -abend google-nested:${{ matrix.system }}:tests/nested/...
     - name: Cache successful run

HACKING.md

@@ -222,10 +222,55 @@ maybe you need to replace system installed snap-seccomp with the one aligned to
 you are testing. To do this, simply backup /usr/lib/snapd/snap-seccomp and overwrite it with 
 the testing one. Don't forget to rollback to the original when finish testing)
 
+### Running nested tests
+
+Nested tests are used to validate features which cannot be tested on regular tests.
+
+The nested test suites work different from the other test suites in snapd. In this case each test runs in a new image
+which is created following the rules defined for the test.
+
+The nested tests are executed using spread tool. See the following examples using the qemu and google backends.
+
+    . `qemu: spread qemu-nested:ubuntu-20.04-64:tests/nested/core20/tpm`
+    . `google: spread google-nested:ubuntu-20.04-64:tests/nested/core20/tpm`
+
+The nested system in all the cases is selected based on the host system. The folloing lines show the relation between host and nested systemd (same applies for classic nested tests):
+
+    . ubuntu-16.04-64 => ubuntu-core-16-64
+    . ubuntu-18.04-64 => ubuntu-core-18-64
+    . ubuntu-20.04-64 => ubuntu-core-20-64
+
+The tools used for creating and hosting the nested vms are:
+
+    . ubuntu-image snap is used to building the images
+    . QEMU is used for the virtualization (with kvm acceleration)
+
+Nested test suite is composed by the following 4 suites:
+
+    classic: the nested suite contains an image of a classic system downloaded from cloud-images.ubuntu.com 
+    core: it tests a core nested system and the images are generated by using ubuntu-image snap
+    core20: this is similar to core suite but tests on it are focused on UC20
+    manual: tests on this suite create a non generic image with spedific conditions
+
+The nested suites use some environment variables to configure the suite and the tests inside it, the most important ones are the described bellow:
+
+    NESTED_WORK_DIR: It is path to the directory where all the nested assets and images are stored
+    NESTED_TYPE: Use core for ubuntu core nested systems or classic instead.
+    NESTED_CORE_CHANNEL: The images are created using ubuntu-image snap, use it to define the default branch
+    NESTED_CORE_REFRESH_CHANNEL: The images can be refreshed to a specific channel, use it to specify the channel
+    NESTED_USE_CLOUD_INIT: Use cloud init to make initial system configuration instead of user assertion
+    NESTED_ENABLE_KVM: Enable kvm in the qemu command line
+    NESTED_ENABLE_TPM: re boot in the nested vm in case it is supported (just supported on UC20)
+    NESTED_ENABLE_SECURE_BOOT: Enable secure boot in the nested vm in case it is supported (just supported on UC20)
+    NESTED_BUILD_SNAPD_FROM_CURRENT: Build and use either core or snapd snapd from current branch
+    NESTED_CUSTOM_IMAGE_URL: Download and use an custom image from this url
+
+
 # Quick intro to hacking on snap-confine
 
 Hey, welcome to the nice, low-level world of snap-confine
 
+
 ## Building the code locally
 
 To get started from a pristine tree you want to do this:

tests/lib/hotplug.sh

@@ -5,36 +5,36 @@
 
 
 hotplug_add_dev1() {
-    add_tty_chardev my-chardev1 /tmp/serialport1
-    add_usb_serial_device my-usb-serial my-chardev1 1234
+    nested_add_tty_chardev my-chardev1 /tmp/serialport1
+    nested_add_usb_serial_device my-usb-serial my-chardev1 1234
 }
 
 hotplug_del_dev1() {
-    del_device my-usb-serial
-    remove_chardev my-chardev1
+    nested_del_device my-usb-serial
+    nested_remove_chardev my-chardev1
 }
 
 hotplug_add_dev2() {
-    add_tty_chardev my-chardev2 /tmp/serialport2
-    add_usb_serial_device my-usb-serial2 my-chardev2 5678
+    nested_add_tty_chardev my-chardev2 /tmp/serialport2
+    nested_add_usb_serial_device my-usb-serial2 my-chardev2 5678
 }
 
 hotplug_del_dev2() {
-    del_device my-usb-serial2
-    remove_chardev my-chardev2
+    nested_del_device my-usb-serial2
+    nested_remove_chardev my-chardev2
 }
 
 # Check that given slot is not present in 'snap connections' output
 # (note, it can still be present in the state with hotplug-gone=true)
 check_slot_not_present() {
     SLOT_NAME="$1"
     for _ in $(seq 10); do
-        if ! execute_remote "snap connections system" | MATCH ":$SLOT_NAME"; then
+        if ! nested_exec "snap connections system" | MATCH ":$SLOT_NAME"; then
             break
         fi
         sleep 1
     done
-    if execute_remote "snap connections system" | MATCH ":$SLOT_NAME "; then
+    if nested_exec "snap connections system" | MATCH ":$SLOT_NAME "; then
         echo "slot $SLOT_NAME shouldn't be present anymore"
         exit 1
     fi
@@ -44,65 +44,65 @@ check_slot_not_present() {
 check_slot_present() {
     SLOT_NAME="$1"
     for _ in $(seq 10); do
-        if execute_remote "snap connections system" | MATCH "serial-port .* - .* :$SLOT_NAME"; then
+        if nested_exec "snap connections system" | MATCH "serial-port .* - .* :$SLOT_NAME"; then
             break
         fi
         sleep 1
     done
-    execute_remote "snap connections system" | MATCH "serial-port .* - .* :$SLOT_NAME"
+    nested_exec "snap connections system" | MATCH "serial-port .* - .* :$SLOT_NAME"
 }
 
 # Check that given slot has hotplug-gone=true, meaning the device was unplugged but there are connections remembered for it
 check_slot_gone() {
     SLOT_NAME="$1"
-    execute_remote 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"hotplug-gone\"]" /var/lib/snapd/state.json' | MATCH "true"
+    nested_exec 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"hotplug-gone\"]" /var/lib/snapd/state.json' | MATCH "true"
 }
 
 # Check that given slot has hotplug-gone=false, meaning the device is plugged
 check_slot_not_gone() {
     SLOT_NAME="$1"
-    execute_remote 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"hotplug-gone\"]" /var/lib/snapd/state.json' | MATCH "false"
+    nested_exec 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"hotplug-gone\"]" /var/lib/snapd/state.json' | MATCH "false"
 }
 
 # Check that given slot has no record in "hotplug-slots" map in the state
 check_slot_not_present_in_state() {
     SLOT_NAME="$1"
-    execute_remote 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"] // \"missing\"" /var/lib/snapd/state.json' | MATCH "missing"
+    nested_exec 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"] // \"missing\"" /var/lib/snapd/state.json' | MATCH "missing"
 }
 
 check_slot_device_path() {
     SLOT_NAME="$1"
     DEVICE_PATH="$2"
-    execute_remote 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"static-attrs\"].path" /var/lib/snapd/state.json' | MATCH "$DEVICE_PATH"
+    nested_exec 'sudo jq -r ".data[\"hotplug-slots\"][\"'"$SLOT_NAME"'\"][\"static-attrs\"].path" /var/lib/snapd/state.json' | MATCH "$DEVICE_PATH"
 }
 
 # Check that given slot is connected to the serial-port-hotplug snap, per 'snap connections' output
 check_slot_connected() {
     SLOT_NAME="$1"
     for _ in $(seq 10); do
-        if execute_remote "snap connections" | MATCH "serial-port .*serial-port-hotplug:serial-port .*$SLOT_NAME"; then
+        if nested_exec "snap connections" | MATCH "serial-port .*serial-port-hotplug:serial-port .*$SLOT_NAME"; then
             break
         fi
         sleep 1
     done
-    execute_remote "snap connections" | MATCH "serial-port .*serial-port-hotplug:serial-port .*$SLOT_NAME"
+    nested_exec "snap connections" | MATCH "serial-port .*serial-port-hotplug:serial-port .*$SLOT_NAME"
 }
 
 # Check that apparmor profile allows rw access to given device path.
 verify_apparmor_profile() {
     DEVPATH=$1
     for _ in $(seq 10); do
-        if execute_remote "cat /var/lib/snapd/apparmor/profiles/snap.serial-port-hotplug.consumer" | MATCH "$DEVPATH rwk,"; then
+        if nested_exec "cat /var/lib/snapd/apparmor/profiles/snap.serial-port-hotplug.consumer" | MATCH "$DEVPATH rwk,"; then
             break
         fi
         sleep 1
     done
-    execute_remote "cat /var/lib/snapd/apparmor/profiles/snap.serial-port-hotplug.consumer" | MATCH "$DEVPATH rwk,"
+    nested_exec "cat /var/lib/snapd/apparmor/profiles/snap.serial-port-hotplug.consumer" | MATCH "$DEVPATH rwk,"
 }
 
 wait_for_all_changes() {
     for _ in $(seq 10); do
-        if ! execute_remote "snap changes" | MATCH "Doing"; then
+        if ! nested_exec "snap changes" | MATCH "Doing"; then
             break
         fi
         sleep 1