Running IoT Edge from within a container

[jsrc27]

I realize this is a bit of a niche use case but, perhaps someone here may be able to help or give me a hint on how to proceed here. In summary, as the title says I'm trying to run the IoT edge daemon from within a container, rather than on the host. In the past my colleagues and I were able to do this with something like this: https://github.com/VMinute/torizon-azure-iot-edge/tree/master

Though this was on a now old version of IoT edge, and I'm trying to do something similar with the current version.

For context I'm running on an ARM64 embedded hardware based on the NXP i.MX8. The OS is a custom embedded OS created with Yocto, this OS does have Docker. I realize there is a meta-layer for IoT Edge but, again this is just an experiment to see if I can containerize this. Also if this is just doomed to not work, then that would be good to know as well.

Steps to Reproduce

1. I created a container image via the following Dockerfile:

FROM ubuntu:latest

RUN apt update && apt install -y systemd wget init sudo

RUN yes 'root' | passwd root
RUN groupadd docker
RUN usermod -a -G docker root

RUN wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
RUN dpkg -i packages-microsoft-prod.deb
RUN apt update && apt install -y aziot-edge

CMD ["/sbin/init"]

2. Run this container image like so: docker run --privileged --rm -it --network=host --tmpfs /run --tmpfs /tmp -v /usr/bin/docker:/usr/bin/docker -v /var/run/docker.sock:/var/run/docker.sock aziot
3. Now within this container image I manually provision with iotedge config mp --connection-string '<PRIMARY CONNECTION STRING> and then iotedge config apply -c '/etc/aziot/config.toml'
4. I then check the state of things with iotedge system status and there are issues. (more details below)

Context (Environment)

Output of iotedge check

Click here

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"

    Caused by:
        parameter "id" has an invalid value
        caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (ConnectionRefused) : Connection refused /var/run/iotedge/mgmt.sock
    One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeAgent container
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
× container on the default network can connect to upstream AMQP port - Error
    Container on the default network could not connect to test-toradex.azure-devices.net:5671
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
× container on the IoT Edge module network can connect to upstream AMQP port - Error
    Container on the azure-iot-edge network could not connect to test-toradex.azure-devices.net:5671
× container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error
    Container on the azure-iot-edge network could not connect to test-toradex.azure-devices.net:443
26 check(s) succeeded.
2 check(s) raised warnings. Re-run with --verbose for more details.
7 check(s) raised errors. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

• Host OS: As said above the actual host OS is a custom Yocto-built Linux. But the container itself is based on Ubuntu 22.04.3
• Architecture: ARM64
• Container OS: Linux containers

Runtime Versions

• aziot-edged: iotedge 1.4.20
• Edge Agent: Even though this container never seems to start I can see version 1.4 gets pulled onto my device
• Edge Hub: System doesn't get to this point
• Docker/Moby: 20.10.25-ce

Logs

aziot-edged logs

Nov 29 22:54:43 apalis-imx8-06738453 systemd[1]: Started Azure IoT Edge daemon.
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Starting Azure IoT Edge Daemon
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Version - 1.4.20
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Obtaining Edge device provisioning data...
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Device is test on test-toradex.azure-devices.net
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Initializing module runtime...
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [INFO] - Using runtime network id azure-iot-edge
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [WARN] - container runtime error
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: Caused by:
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]:     0: error trying to connect: Permission denied (os error 13)
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]:     1: Permission denied (os error 13)
Nov 29 22:54:43 apalis-imx8-06738453 aziot-edged[1357]: 2023-11-29T22:54:43Z [ERR!] - Failed to initialize module runtime: runtime operation error: initialize module runtime
Nov 29 22:54:43 apalis-imx8-06738453 systemd[1]: aziot-edged.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 22:54:43 apalis-imx8-06738453 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.

[arsing]

keyd, certd and identityd authenticate their clients via their UIDs, which won't work if the client isn't in the same userns.

edged's workload socket authenticates its clients via their PIDs, which won't work if the client isn't in the same pidns.

A docker-in-docker setup, ie a setup where all i-i-s and iotedge services run in the same container and a nested dockerd runs in the same container, might work. The setup you tried, with dockerd running on the host, won't.

---

In the past my colleagues and I were able to do this with something like this: https://github.com/VMinute/torizon-azure-iot-edge/tree/master

This worked because you configured iotedged to use TCP instead of UDS for the workload socket, in which case iotedged did not do any authentication. This had always been a "not-for-production" feature (it used to be needed for Windows before Windows gained its own UDS support), and was eventually removed. It won't work in 1.4

[jsrc27]

A docker-in-docker setup, ie a setup where all i-i-s and iotedge services run in the same container and a nested dockerd runs in the same container, might work. The setup you tried, with dockerd running on the host, won't.

Well, I appreciate you telling me straight away this approach I'm trying won't work as is. Saves me quite a bit of time experimenting on this approach.

I did consider a docker-in-docker setup. I just thought it might be clunky if a container/module needs access to something on the host like files, or hardware peripherals, or such. I suppose I could pass this into the docker-in-docker container then from there into the needed containers. Not ideal, but still a path forward perhaps.

In any case thanks again for the input, I'm aware this is quite a niche use-case.

[nyanzebra]

@jsrc27 are you okay with this issue being resolved? Want to make sure all questions were addressed :)

[jlian]

Closing for now, let us know if you have additional questions and we can reopen

[Clockwork-Muse]

Where it might not be a niche use-case:

I would like to do integrated module tests without the need to spin up an IoTHub in Azure, or register devices with the hub, for development/testing scenarios.
Partially this is to support offline build/test setups, but also just because it simplifies developer setup, including having to worry about potential online resource conflicts.