Network reliability issues

[elventear]

I recently became a new user of lima, via colima, while looking for an alternative of docker-desktop on the Mac.

As I have been adopting the tool to my development workflow I have encountered issues where the network inside the VM seems to stall for long periods of time; this is visible as network operations are issued inside the VM leading to timeouts and lack of connectivity. I spent some time scouring bug reports and documents to try to find a workaround. Here are some of my findings (everything to be reported was tested with MacPorts lima 0.8.3 and macOS 12.2.1):

1. As the SLIRP user mode networking seems to be root cause of the problem Network in containers breaks under bigger network load abiosoft/colima#140, I have tried to use vde_vmnet. The first thing I have noticed is that when enabling use of the alternate network modes, the SLIRP network is still created and has a higher precedence as default route in the routing table, which means that default network activity is going to still flow via the SLIRP network and be subject the stalling issues.
2. As part of the provisioning step, I remove the default route going via eth0 and this solved the previous problem, but seems the vde_switch based connectivity is really slow as reported by Networking speeds are slow, is this expected? vde_vmnet#26.
3. Since I was seeing the vde2 problems as reported in [vde_switch] No buffer space available vde_vmnet#19 and seeing there is a new patch related in virtualsquare!vde-2#35, I decided to upgrade vde-2 to virtualsquare/vde-2@506ce85. This seemed to improve speeds, but I still could see timeouts happening within my VM.
4. Instead of using switch mode, I switched to PTP mode and this seems to have solved most of the reliability and speed problems. But adds the question of how to manage the vde_vmnet process seamlessly.

A few questions:

• Since limactl is already able to start vde_switch and vde_vmnet when using host, shared or bridged networking, is it possible to also start vde_vmnet for the PTP mode? It becomes cumbersome to have to manage vde_vmnet manually for every VM we want to boot.
• When using an alternate networking mode in addition to SLIRP is it possible to remove entirely SLIRP as a default network route automatically?

As a reference, this is the template that I am now using, based off colima:

---
arch: x86_64
images:
  - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.2/alpine-lima-clm-3.14.3-aarch64.iso
    arch: aarch64
    digest: sha512:1f93efbfb3093203446f605647438d04d239d3ee591c06b85e79f067180eff5d6b24b9133d147731a0a68d6d3b8d233eed844c99453e2f697e8df70f5b9cb1eb
  - location: https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.2/alpine-lima-clm-3.14.3-x86_64.iso
    arch: x86_64
    digest: sha512:875482176ff2f43bf9472f84137d8b9a56d692ae19243436079c7f2f78cb6b13576601eef9102aaea796629215fc60308e69b590c81fe4139e9e84f1ec4a57f5
cpus: 8
memory: 4GiB
disk: 100GiB
mounts:
  - location: "~"
    writable: true
  - location: /tmp/colima
    writable: true
ssh:
  localPort: 51397
  loadDotSSHPubKeys: false
containerd:
  system: false
  user: false
firmware:
  legacyBIOS: false
useHostResolver: true
portForwards:
  - guestPortRange:
      - 0
      - 0
    guestSocket: /var/run/docker.sock
    hostPortRange:
      - 0
      - 0
    hostSocket: /Users/elventear/.lima/colima/docker.sock
    proto: tcp
  - guestIP: 127.0.0.1
    guestPortRange:
      - 1
      - 65535
    hostIP: 0.0.0.0
    hostPortRange:
      - 1
      - 65535
    proto: tcp
networks:
  - vnl: /Users/elventear/.lima/_networks/colima.ptp
    switchPort: 65535
provision:
  - mode: system
    script: |
      #!/bin/bash
      sysctl -w net.ipv6.conf.all.disable_ipv6=1
      sysctl -w net.ipv6.conf.default.disable_ipv6=1
      SLIRP_ROUTE="$(ip route | egrep 'default.*eth0')"
      test -n "$SLIRP_ROUTE" && ip route del $SLIRP_ROUTE
      apk add htop tcpdump iftop tmux
      service docker start
      while ! test -e /var/run/docker.sock; do
        sleep 0.5
      done
      chmod ogu+rw /var/run/docker.sock