Add workflow with DTS tests #193
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| --- | ||
| name: Build DTS | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| cacheless: | ||
| type: boolean | ||
| required: true | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: | ||
| labels: dts-builder | ||
| steps: | ||
| - name: Checkout meta-dts repo | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| path: "meta-dts" | ||
| - name: Prepare cache-less build configuration | ||
| if: ${{ inputs.cacheless }} | ||
| shell: bash | ||
| run: | | ||
| sed -i '/cache.yml/d' meta-dts/kas.yml | ||
| - name: Build DTS image | ||
| shell: bash | ||
| id: build_image | ||
| run: | | ||
| for attempt in {1..5}; do | ||
| if kas-container build meta-dts/kas.yml; then | ||
| echo "Build command succeeded on attempt $attempt" | ||
| break | ||
| else | ||
| echo "Build command failed on attempt $attempt" | ||
| if [ $attempt -lt 5 ]; then | ||
| sleep 5 | ||
| fi | ||
| fi | ||
| done | ||
| continue-on-error: true | ||
| - name: Report build command | ||
| run: | | ||
| if [ ${{ steps.build_image.outcome }} == 'failure' ]; then | ||
| echo "All build attempts failed." | ||
| exit 1 | ||
| else | ||
| echo "At least one build attempt succeeded." | ||
| fi |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| --- | ||
| name: Run DTS tests | ||
| on: | ||
| pull_request: | ||
| branches: | ||
| - 'main' | ||
| jobs: | ||
| build-dts: | ||
| uses: ./.github/workflows/build.yml | ||
| with: | ||
| cacheless: false | ||
| run-tests: | ||
| name: Run DTS tests | ||
| if: ${{ github.head_ref == 'develop' && contains(join(needs.*.result, ','), 'success') }} | ||
| needs: build | ||
| runs-on: | ||
| labels: dts-builder | ||
| steps: | ||
| - name: Checkout OSFV repo | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: 'Dasharo/open-source-firmware-validation' | ||
| path: 'open-source-firmware-validation' | ||
| submodules: 'recursive' | ||
| ref: 'develop' | ||
| - name: Run IPXE server | ||
| shell: bash | ||
| run: | | ||
| mkdir ipxe | ||
| cp build/tmp/deploy/images/genericx86-64/dts-base-image-genericx86-64.cpio.gz ipxe | ||
| cp build/tmp/deploy/images/genericx86-64/bzImage ipxe | ||
| echo -e "\n | ||
| #!ipxe\n | ||
| imgfetch --name file_kernel bzImage\n | ||
| imgfetch --name file_initrd dts-base-image-genericx86-64.cpio.gz\n | ||
| kernel file_kernel root=/dev/nfs initrd=file_initrd\n | ||
| boot" > ipxe/dts.ipxe | ||
| cd ipxe && python3-m http.server 4321 & | ||
|
There was a problem hiding this comment. Are we certain all of these background processes are always killed when workflow is interrupted, or we need to take care of this? This is less of a problem on GitHub runners, as each time new VM is started, but on the self-hosted runner, we may end up with zombie processes if this is not the case. |
||
| - name: Install requirements | ||
| shell: bash | ||
| run: | | ||
| cd open-source-firmware-validation | ||
| python3 -m virtualenv venv | ||
| source venv/bin/activate | ||
| pip install -r requirements.txt | ||
| - name: Run QEMU | ||
| shell: bash | ||
| id: run_qemu | ||
| run: | | ||
| cd open-source-firmware-validation/scripts/ci | ||
| mkdir qemu-data | ||
| touch qemu-data/hdd.qcow | ||
| ./qemu-run.sh nographic os & | ||
| - name: Create directory for logs | ||
| shell: bash | ||
| id: log_dirs | ||
| run: | | ||
| timestamp=$(date -u +%Y-%m-%dT%H:%M:%S%Z) | ||
|
There was a problem hiding this comment.
|
||
| directory="/tmp/dts-test-ci-${timestamp}" | ||
| mkdir $directory | ||
| echo "directory=$directory" >> "$GITHUB_OUTPUT" | ||
| - name: Run tests | ||
| shell: bash | ||
| env: | ||
| LOG_DIR: ${{ steps.log_dirs.outputs.directory }} | ||
| run: | | ||
| cd open-source-firmware-validation | ||
| source venv/bin/activate | ||
| # This file is already present on dts-builder. | ||
| # It contains credentials for DPP subscriptions. | ||
| # It has the following form: | ||
| # DPP_PASSWORD="..." | ||
| # <SUBSCRIPTION_TYPE>_DOWNLOADS="download key" | ||
| # <SUBSCRIPTION_TYPE>_LOGS="logs key" | ||
| # (...) | ||
| source ~/.secrets/dpp-keys | ||
macpijan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ip_addr=$(ip -o -4 addr list eno2 | awk '{print $4}' | cut -d/ -f1) | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
|
There was a problem hiding this comment. Is there a reason we do not use https://github.com/Dasharo/open-source-firmware-validation/blob/develop/scripts/run.sh here? We do not want to publish full logs? But we do not store artifacts anyway? Please explain it here. There was a problem hiding this comment. We don't want to print full logs because in case of a failure they might leak DPP credentials There was a problem hiding this comment. If running these DTS scripts is very different than running There was a problem hiding this comment. But maybe it is not (or does not have to be) and we can simply run the run.sh script? There was a problem hiding this comment.
To the terminal console? Do we log the DPP variables to the console? I know we might have them in HTML log files, but can we have them in the console output as well? There was a problem hiding this comment. We can use https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#masking-a-value-in-a-log to lessen chance that secret information gets logged. |
||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$MSI_DOWNLOAD \ | ||
|
There was a problem hiding this comment. I guess you could shorten execution in CI by using https://robotframework.org/robotframework/latest/libraries/OperatingSystem.html#Get%20Environment%20Variable in tests to retrieve credentials, if we assume to always get these from OS environment |
||
| -v dpp_logs_key:$MSI_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "msi" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_msi.log | grep "| PASS |\|| FAIL |" | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
|
There was a problem hiding this comment. This seems overly complex and scales poorly. Again, what happens if we add another platform/test in the dts-e2e.robot? We need to update the CI file as well - this is not optimal. Why cannot we run the whole suite in one run? We can modify the tests in the suite to set DPP variables in each test case accordingly, based on the environment variables. There was a problem hiding this comment. There was a problem hiding this comment. This will be a problem for a short period of time, because soon, when we migrate our infrastructure to MinIO, we will be able to use one set of credentials for all subscriptions. So, I would rather stay with the way it is now. |
||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$MSI_HEADS_DOWNLOAD \ | ||
| -v dpp_logs_key:$MSI_HEADS_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "msi_heads" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_msi_heads.log | grep "| PASS |\|| FAIL |" | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$OPTIPLEX_DOWNLOAD \ | ||
| -v dpp_logs_key:$OPTIPLEX_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "optiplex" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_optiplex.log | grep "| PASS |\|| FAIL |" | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$NOVACUSTOM_HEADS_DOWNLOAD \ | ||
| -v dpp_logs_key:$NOVACUSTOM_HEADS_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "novacustom_heads" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_nc_heads.log | grep "| PASS |\|| FAIL |" | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$PCENGINES_DOWNLOAD \ | ||
| -v dpp_logs_key:$PCENGINES_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "pcengines" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_pcengines.log | grep "| PASS |\|| FAIL |" | ||
|
|
||
| robot -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v snipeit:no \ | ||
| -v dpp_password:$DPP_PASSWORD -v dpp_download_key:$PCENGINES_SEABIOS_DOWNLOAD \ | ||
| -v dpp_logs_key:$PCENGINES_SEABIOS_LOGS -v netboot_utilities_support:True \ | ||
| -v dts_ipxe_link:http://${ip_addr}:4321/dts.ipxe | ||
| -i "pcengines_seabios" dts/dts-e2e.robot 2>&1 | tee $LOG_DIR/output_pcengines_seabios.log | grep "| PASS |\|| FAIL |" | ||
| - name: Copy log | ||
| shell: bash | ||
| env: | ||
| LOG_DIR: ${{ steps.log_dirs.outputs.directory }} | ||
| run: | | ||
| cp open-source-firmware-validation/log.html $LOG_DIR/log.html | ||
| cleanup: | ||
| name: Cleanup | ||
| if: always() | ||
| needs: run-tests | ||
| runs-on: | ||
| labels: dts-builder | ||
| steps: | ||
| - name: Cleanup after tests | ||
| shell: bash | ||
| run: | | ||
| rm -rf open-source-firmware-validation meta-dts build ipxe | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I would put this into the osfv/scripts/ci as well, just as qemu.
This way, you could replicate this more easily locally as well. I assume you might want to run these tests locally as well, prior creating PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this script to osfv: d46ef63
PR to osfv: Dasharo/open-source-firmware-validation#620
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@PLangowski please check
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@PLangowski updated script in the osfv PR, tomorrow I will update its usage here