Secure boot for XCP-ng platform #294

olivierlambert · 2019-10-17T13:16:30Z

Context

Security starts on the hardware where the system boots. You need to "trust" your system, it's a "chain of trust". When XCP-ng boots, we should be able to boot it with secure boot enabled.

Suggestions

This is a complicated topic, but benefits are huge. A first step is xen.efi able to secure boot using Shim.

The text was updated successfully, but these errors were encountered:

stormi · 2019-10-17T13:17:01Z

You mean support for Secure boot when installing XCP-ng, or for guests?

olivierlambert · 2019-10-17T13:17:42Z

For XCP-ng itself, on platform boot.

WSLUser · 2020-02-27T15:33:37Z

Any progress with this? It's a requirement to run UEFI + Secure Boot on some government systems. Currently you have to accept the risk if you use Xen. Can you boost priority?

olivierlambert · 2020-02-27T15:52:11Z

We have a dev coming in one month that will be dedicated to this feature.

Ceaus · 2020-12-30T19:40:16Z

Just to add: the ODROID-H2+ only supports secure boot, so currently can't install XCP-ng from bootable usb. I think I have no other option than to wait for XCP-ng secure boot support.

olivierlambert · 2020-12-30T19:58:18Z

Hmm there's a Xen bug on ODROID-H2 preventing it to boot. It's not related to secure boot IMHO.

edit: for the ODROID-H2+.

rjt · 2020-12-30T20:48:47Z

When secure boot is enabled, will XCP-ng then support the existing XEN vTPM virtual TPM virtual machine? So when the hypervisor boots up on hardware with TPM, the vTPM machine is one of the first to boot afterwards and manages keys from the hardware TPM for each VM? So simpler encrypted drives for VMs.

…

On Wed, Dec 30, 2020 at 1:58 PM Olivier Lambert ***@***.***> wrote: Hmm there's a Xen bug on ODROID-H2 preventing it to boot. It's not related to secure boot IMHO. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACX7F4SRN56QXJ7DNZXQNTSXOA6PANCNFSM4JBZOOWQ> .

olivierlambert · 2020-12-30T21:20:16Z

It's a question for @beshleman

beshleman · 2020-12-30T21:30:42Z

@rjt That is something we can look into supporting, although the vTPM is orthogonal to UEFI Secure Boot, and could be accomplished without UEFI Secure Boot AFAIK.

I just created the this issue to record your request.

This issue (#294) involves changing the multiboot2 Xen binary to allow being measured and verified by grub / shim, as well as implementing in Xen verification of the dom0 kernel using the shim MOKList.

Ceaus · 2020-12-30T22:01:57Z

Hmm there's a Xen bug on ODROID-H2 preventing it to boot. It's not related to secure boot IMHO.

Perhaps I've searched the issues not good enough. I didn't see anything specific to the H2+.

In any case I've tried various bootable USB images today on the Odroid H2+. Secure boot images boot up straight away. Legacy BIOS images are halting immediately after turning on the H2+ with the cursor blinking at the left top corned. And that includes the images from XCP-ng.

Going into the BIOS setup the USB stick is listed as a bootable device if the image is an UEFI (?) image. If the image is a legacy boot image the USB stick is NOT listed as a bootable device. So I do think that this is about the XCP-ng image itself, not having the required security settings.

nagilum99 · 2020-12-30T22:22:33Z

That would require that autostart-function would need to learn startup priorities. Currently (unless I missed an update) it's started ~simultaneously/random.
Via vApp you can set priorities and waiting times, though it doesn't automatically start the vApp list.

A solution here could be that XCP-ng would autostart a (maybe specially named) vApp group. I'm not aware that vApps are going to be depricated anytime soon, so it could be a solid way without too much code diff to CH.
(If I'm right, enabling Autostart VMs in XCP changes something like a rc.local file, that runs last and thereby it could bump xapi to start a vApp instead of (direct) VMs.

...that would also help people using Windows AD to start their DC(s) first etc.

beshleman · 2021-01-08T03:19:57Z

It'll be important to build grub with the load/iorw/memrw modules disabled. We'll need to audit other modules for compromising functionality.

beshleman · 2021-01-20T19:46:23Z

Just an update here: the Xen patch set is complete and has passed (very) preliminary testing.

Putting some final touches on it before hitting the mailing list with it.

This is just for making Xen capable. Following acceptance upstream, we'll need to:

Obtain a signed shim from Microsoft for XCP-ng
Rebuild and sign kernel, grub, xen and shim.
Release RPM for shim and new signed binaries.
Modify the installer

WSLUser · 2021-07-23T16:06:40Z

@beshleman Anything to report here?

beshleman · 2021-07-23T19:15:17Z

@beshleman Anything to report here?

For a basic chain-of-trust boot chain, I have a working PoC. The verification chain successfully extends from the firmware, through Xen, and into the dom0 kernel. I'm not expecting a production release until next year, though, because there is a significant amount of work needed in both the dom0 kernel and hypervisor in order to harden the system against runtime attacks that subvert secure boot protections.

If interested in the details, my Xen Summit talk goes into what we're looking at: https://youtu.be/A_IhKjK7EgA

rjt · 2021-07-23T21:00:09Z

I am a total noob and thank you for developing. Now that Windows 11 requires a TPM, would think Win11 VMs need a vTPM. Haven’t tested. Wasnt the case 6 months ago. Will follow up on the vTPM github issue. #471

…

On Fri, Jul 23, 2021 at 1:15 PM beshleman ***@***.***> wrote: @beshleman <https://github.com/beshleman> Anything to report here? For a basic chain-of-trust boot chain, I have a working PoC. The verification chain successfully extends from the firmware, through Xen, and into the dom0 kernel. I'm not expecting a production release until next year, though, because there is a significant amount of work needed in both the dom0 kernel and hypervisor in order to harden the system against runtime attacks that subvert secure boot protections. If interested in the details, my Xen Summit talk goes into what we're looking at: https://youtu.be/A_IhKjK7EgA — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACX7F5WTFF3W5XVM7SHKN3TZG5VDANCNFSM4JBZOOWQ> .

pietrushnic · 2021-08-01T23:51:47Z

On Tuesday as part of Qubes OS mini-summit we will sum up effort and complexity of S-RTM and Secure Boot for VMs and not only, feel free to join: https://www.qubes-os.org/news/2021/07/30/minisummit-agenda/

connorc0405 · 2022-07-07T01:04:25Z

@beshleman do you have any instructions for self-signing to enable secure boot?

beshleman · 2022-07-07T01:21:08Z

@beshleman do you have any instructions for self-signing to enable secure boot?

For what it is worth, that is probably just security theater until Xen + grub is secure boot capable.

But if you are just experimenting and aren't concerned with production-ready security, then I think you can do it with new versions of grub, the chainload command, and a signed Xen.efi... but it's been a while since I tried so could be leading you astray.

m-iwanicki · 2025-02-04T16:29:29Z

I managed to get working PoC as part of Xen Winter Meetup 2025 preparations. More about this can be found at: https://cfp.vates.tech/xen-meetup-2025/talk/8JBQKC/

I took similar approach when enabling SB in Qubes OS (more can be read on QubesOS/qubes-issues#8206 (comment))

If anyone is interested:

To make booting XCP-ng with enabled SB possible you have to build xen.efi first (as it isn't packaged in xen-hypervisor-*.rpm). To do that I just followed steps in https://docs.xcp-ng.org/project/development-process/local-rpm-build/ and built xen from https://github.com/xcp-ng-rpms/xen. I used file under BUILD/xen-4.17.5/xen/build-xen-release/xen.efi
Download GRUB modules from https://updates.xcp-ng.org/8/8.3/base/x86_64/Packages/grub-efi-2.06-4.0.2.xcpng8.3.x86_64.rpm and unpack them
```
rpm2cpio grub-efi-*.rpm | cpio -imvd
```
You could also use modules available on your system (if you are using XCP-ng)

Create sbat.csv containing e.g.

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
grub,3,Free Software Foundation,grub,2.06,https//www.gnu.org/software/grub/

Build GRUB (I only added minimal amount of GRUB modules):

GRUB_MODULES="boot reboot chain fat font gettext ls part_msdos part_gpt serial search multiboot2 ext2 multiboot gzio cpio"
grub-mkimage -d usr/lib/grub/x86_64-efi -o grubx64.efi -O x86_64-efi -p "/EFI/xenserver/" --sbat sbat.csv ${GRUB_MODULES}

or grub2-mkimage depending on OS.

Sign it with custom key (you should enroll your custom certificate to DB)
Copy signed grubx64.efi to /boot/efi/EFI/BOOT/
Build shim, sign shimx64.efi and mmx64.efi and copy mmx64.efi to /boot/efi/EFI/BOOT/ and shimx64.efi to /boot/efi/EFI/BOOT/BOOTX64.efi (overwrite existing file)
Create empty config: dd if=/dev/zero of=config bs=1 count=1 (can be used to embed config instead of passing arguments via chainloader)

Create UKI file containing Xen, dom0 kernel and initramfs. To do this I used https://github.com/QubesOS/qubes-core-admin-linux/blob/main/uki-generate script.

./uki-generate xen.efi config /boot/vmlinuz-4.19-xen /boot/initrd-4.19-xen.img uki.efi

It basically boils down to

objcopy --section-alignment=0x200000 --remove-section=.gnu.* \
    --remove-section=.buildid --strip-debug \
    --add-section=.config=config \
    --change-section-vma=.config=0xffff82d042000000 \
    --add-section=.kernel=/boot/vmlinuz-4.19-xen \
    --change-section-vma=.kernel=0xffff82d042200000 \
    --add-section=.ramdisk=/boot/initrd-4.19-xen.img \
    --change-section-vma=.ramdisk=0xffff82d042a00000 \
    -- ./xen.efi uki.efi

Sign uki.efi and put it in /boot/efi/EFI/xenserver/

Modify /boot/efi/EFI/xenserver/grub.cfg. Example config (fresh installation on QEMU):

[16:40 xcp-ng ~]# cat /boot/efi/EFI/xenserver/grub.cfg
serial --unit=0 --speed=115200
terminal_input serial console
terminal_output serial console
set default=0
set timeout=5

menuentry 'XCP-ng UKI' {
        chainloader /EFI/xenserver/uki.efi placeholder -- dom0_mem=1232M,max:1232M watchdog ucode=scan dom0_max_vcpus=1-4 crashkernel=256M,below=4G console=com1 com1=115200,8n1 -- root=LABEL=root-kxbhxu ro nolvm hpet=disable console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles
}
menuentry 'XCP-ng' {
        search --label --set root root-kxbhxu
        multiboot2 /boot/xen.gz dom0_mem=1232M,max:1232M watchdog ucode=scan dom0_max_vcpus=1-4 crashkernel=256M,below=4G console=vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-kxbhxu ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles
        module2 /boot/initrd-4.19-xen.img
}

I added XCP-ng UKI menuentry with chainloader command in format: chainloader <uki_file> placeholder -- <xen_cmdline> -- <dom0_kernel_cmdline>. <xen_cmdline> was copied from multiboot2 command in XCP-ng menuentry and <dom0_kernel_cmdline> was copied from module2 command.

After that you should be able to enable Secure Boot and boot into XCP-ng.
You can check Secure Boot state by checking dmesg:

[16:46 xcp-ng ~]# dmesg | grep -i secure
[    0.000000] UEFI Secure Boot is enabled.
[    0.297235] Secure boot enabled

I tested this PoC on QEMU:

qemu-system-x86_64 -enable-kvm -smp 4 -m 4G -M q35,kernel-irqchip=split --nographic \
    -global ICH9-LPC.disable_s3=1 \
    -drive if=pflash,format=raw,file=OVMF_CODE.secboot.fd \
    -drive if=pflash,format=raw,file=OVMF_VARS.secboot.fd \
    -drive file=xcp.img,index=0,format=raw

Of course it's not perfect. One issue I found is that when Secure Boot is disabled you can't boot XCP-ng UKI anymore:

Xen 4.17.5-4 (c/s 430ce6cd9365, pq f09cf5633773) EFI loader
Using builtin config file
kernel: 0x000000005a200000-0x000000005a894630
ramdisk: 0x000000005aa00000-0x000000005bcf0a15
Dom0 kernel image could not be verified: ErrCode: 0x8000000000000003

Edit: fixed couple typos/mistakes

pietrushnic · 2025-02-04T23:09:46Z

@m-iwanicki IIUC another issue which is also there is the fact that we cannot get rid of GRUB2 the same way as we saw here?

m-iwanicki · 2025-02-05T11:40:14Z

@pietrushnic Yes:

Xen 4.17.5-4 (c/s 430ce6cd9365, pq f09cf5633773) EFI loader
Unsupported relocation type

Though we can get rid of shim and just boot UKI file directly.
And weirdly, booting directly (without shim/GRUB) works with SB enabled and disabled.

olivierlambert changed the title ~~Secure boot and XCP-ng~~ Secure boot for XCP-ng platform Oct 17, 2019

stormi added the new feature 🆕 Create of a new feature in XCP-ng label Apr 20, 2020

beshleman self-assigned this Apr 20, 2020

beshleman mentioned this issue Dec 30, 2020

Support vTPM for guests #471

Open

beshleman mentioned this issue Jan 26, 2021

Support Xen-specific lockdown for secure boot #478

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secure boot for XCP-ng platform #294

Secure boot for XCP-ng platform #294

olivierlambert commented Oct 17, 2019 •

edited

Loading

stormi commented Oct 17, 2019

olivierlambert commented Oct 17, 2019

WSLUser commented Feb 27, 2020

olivierlambert commented Feb 27, 2020 •

edited

Loading

Ceaus commented Dec 30, 2020 •

edited

Loading

olivierlambert commented Dec 30, 2020 •

edited

Loading

rjt commented Dec 30, 2020 via email

olivierlambert commented Dec 30, 2020

beshleman commented Dec 30, 2020 •

edited

Loading

Ceaus commented Dec 30, 2020

nagilum99 commented Dec 30, 2020

beshleman commented Jan 8, 2021

beshleman commented Jan 20, 2021 •

edited

Loading

WSLUser commented Jul 23, 2021

beshleman commented Jul 23, 2021

rjt commented Jul 23, 2021 via email

pietrushnic commented Aug 1, 2021

connorc0405 commented Jul 7, 2022

beshleman commented Jul 7, 2022

m-iwanicki commented Feb 4, 2025 •

edited

Loading

pietrushnic commented Feb 4, 2025

m-iwanicki commented Feb 5, 2025 •

edited

Loading

Secure boot for XCP-ng platform #294

Secure boot for XCP-ng platform #294

Comments

olivierlambert commented Oct 17, 2019 • edited Loading

Context

Suggestions

stormi commented Oct 17, 2019

olivierlambert commented Oct 17, 2019

WSLUser commented Feb 27, 2020

olivierlambert commented Feb 27, 2020 • edited Loading

Ceaus commented Dec 30, 2020 • edited Loading

olivierlambert commented Dec 30, 2020 • edited Loading

rjt commented Dec 30, 2020 via email

olivierlambert commented Dec 30, 2020

beshleman commented Dec 30, 2020 • edited Loading

Ceaus commented Dec 30, 2020

nagilum99 commented Dec 30, 2020

beshleman commented Jan 8, 2021

beshleman commented Jan 20, 2021 • edited Loading

WSLUser commented Jul 23, 2021

beshleman commented Jul 23, 2021

rjt commented Jul 23, 2021 via email

pietrushnic commented Aug 1, 2021

connorc0405 commented Jul 7, 2022

beshleman commented Jul 7, 2022

m-iwanicki commented Feb 4, 2025 • edited Loading

pietrushnic commented Feb 4, 2025

m-iwanicki commented Feb 5, 2025 • edited Loading

olivierlambert commented Oct 17, 2019 •

edited

Loading

olivierlambert commented Feb 27, 2020 •

edited

Loading

Ceaus commented Dec 30, 2020 •

edited

Loading

olivierlambert commented Dec 30, 2020 •

edited

Loading

beshleman commented Dec 30, 2020 •

edited

Loading

beshleman commented Jan 20, 2021 •

edited

Loading

m-iwanicki commented Feb 4, 2025 •

edited

Loading

m-iwanicki commented Feb 5, 2025 •

edited

Loading