Port forwarding doesn't work for non-127.0.0.1 localhost

[matthew-nichols]

Description

nerdctl run --rm -p 127.0.0.2:8080:80 nginx does not work as expected:

$ curl 127.0.0.2:8080
curl: (56) Recv failure: Connection reset by peer

nerdctl run --rm -p 127.0.0.1:8080:80 nginx does work, and outputs the appropriate nginx welcome page from curl 127.0.0.1:8080

Can confirm (on another Linux box) that the same functionality (listen on 127.0.0.2:8080) works in rootless Podman.

Note: the output for nerdctl port in general, seems reversed to what I would expect but contains the right address:

$ nerdctl port nginx
80/tcp -> 127.0.0.2:8080
$ nerdctl port nginx
80/tcp -> 127.0.0.1:8080

ss also confirms something is listening on the right address:

$ sudo ss -tupln | grep 8080
tcp   LISTEN 0      4096            127.0.0.2:8080       0.0.0.0:*    users:(("rootlesskit",pid=1820,fd=30))

Steps to reproduce the issue

1.Install and setup rootless nerdctl
2.Run nerdctl run --rm -p 127.0.0.2:8080:80 nginx
3.Attempt curl 127.0.0.2:8080
4.Get reset by peer

Describe the results you received and expected

Received: reset by peer
Expected: to be connected to the service in the container

What version of nerdctl are you using?

nerdctl version 2.0.0-rc.2, rootless, Linux (Pop!_OS 22.04 LTS), install from tar

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

$ nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v2.0.0-rc.4
 Storage Driver: overlayfs
 Logging Driver: json-file
  Cgroup Driver:  : systemd
  Cgroup Version: : 2
 Plugins:
  Log:     fluentd journald json-file syslog
  Storage: native overlayfs stargz
 Security Options:
  apparmor
  seccomp
   Profile:	builtin
  cgroupns
  rootless
 Kernel Version:   6.9.3-76060903-generic
 Operating System: Pop!_OS 22.04 LTS
 OSType:           linux
 Architecture:     x86_64
 CPUs:             16
 Total Memory:     30.56GiB
 Name:             hostname
 ID:               2eb24a77-0449-485d-b84a-e18aca278b1e

WARNING: AppArmor profile "nerdctl-default" is not loaded.
         Use 'sudo nerdctl apparmor load' if you prefer to use AppArmor with rootless mode.
         This warning is negligible if you do not intend to use AppArmor.
WARNING: No cpu cfs period support
WARNING: No cpu cfs quota support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[apostasie]

@AkihiroSuda tag expert?

From a glance, I am not sure this can be solved inside nerdctl.

[AkihiroSuda]

This is probably a bug of https://github.com/rootless-containers/rootlesskit.

Workarounds:

1. Set Environment="CONTAINERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns" https://rootlesscontaine.rs/getting-started/containerd/#changing-the-port-forwarder
2. Or, use rootful mode (sudo nerdctl)

[fahedouch]

This is probably a bug of https://github.com/rootless-containers/rootlesskit.

Workarounds:

1. Set Environment="CONTAINERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns" https://rootlesscontaine.rs/getting-started/containerd/#changing-the-port-forwarder
2. Or, use rootful mode (sudo nerdctl)

it is not bug. The default ROOTLESSKIT_PORT_DRIVER is builtin which limits network functionality by not propagating source IP addresses. Can we consider slirp4netns as default driver which provide more realistic network behavior ?

[haytok]

Hi, @AkihiroSuda @fahedouch

In the current implementation, when --port-driver is specified as buildin, Rootlesskit Parent NetNS accept() external connections for the address or 127.0.0.1 and port number specified by -p.

On the other hand, Rootlesskit Child NetNS will send the request to 127.0.0.1 regardless of the ip by accept() in Parent NetNS.

• https://github.com/rootless-containers/rootlesskit/blob/master/pkg/port/builtin/child/child.go#L123C1-L158C3

However, that is the expected behavior as described in the document below.

• https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md

builtin 30.0 Gbps Always 127.0.0.1
The builtin driver is fast, but be aware that the source IP is not propagated and always set to 127.0.0.1.

Question

I think propagation of the ip that is accept() in Parent NetNS to Child NetNS will solve this problem, so I'll create a PR in rootlesskit for propagation of parent ip to improve the behavior of this issue. What do you think?

[AkihiroSuda]

Thanks, does your PR pass the nerdctl CI?
LGTM then.
(The proper LGTM will be after reviewing the actual code)

[fahedouch]

Thanks for your efforts but I'm not sure how you plan to forward traffic from a child network namespace to the host network while preserving the source IP address when using the builtin driver with detached network namespace

[haytok]

Thanks for comments !!!

does your PR pass the nerdctl CI?

When I locally apply the modification that is trying to create a PR, curl 127.0.0.2:8080 confirms that it succeeds.

Details

[ec2-user@ip-172-31-40-91 rootlesskit]$ make && sudo make install
make: Nothing to be done for 'all'.
install -D -m 755 /home/ec2-user/rootlesskit/bin/rootlesskit /usr/local/bin/rootlesskit
install -D -m 755 /home/ec2-user/rootlesskit/bin/rootlessctl /usr/local/bin/rootlessctl
install -D -m 755 /home/ec2-user/rootlesskit/bin/rootlesskit-docker-proxy /usr/local/bin/rootlesskit-docker-proxy
[ec2-user@ip-172-31-40-91 rootlesskit]$ sudo systemctl restart user@1000.service
[ec2-user@ip-172-31-40-91 rootlesskit]$ n rm -f nginx && nerdctl run -d --name nginx -p 127.0.0.2:8080:80 nginx && nerdctl ps
nginx
72824134f1fb2d642ce386642f7229316a400ce927025605c4f4decc5cf3a7fd
CONTAINER ID    IMAGE                             COMMAND                   CREATED                   STATUS    PORTS                     NAMES
72824134f1fb    docker.io/library/nginx:latest    "/docker-entrypoint.…"    Less than a second ago    Up        127.0.0.2:8080->80/tcp    nginx
[ec2-user@ip-172-31-40-91 rootlesskit]$ curl 127.0.0.2:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

As for CI, I haven't been able to confirm this yet, so I'll check.

---

I'm not sure how you plan to forward traffic from a child network namespace to the host network while preserving the source IP address when using the builtin driver with detached network namespace

When Parent NetNS sends a request to Child NetNS, the following processing is performed.

• https://github.com/rootless-containers/rootlesskit/blob/master/pkg/port/builtin/parent/tcp/tcp.go#L59C1-L62C62

This spec variable has ParentIP, I think it is enough to propagate it to Child NetNS.

• https://github.com/rootless-containers/rootlesskit/blob/master/pkg/port/port.go#L10C1-L15C49

I haven't written the PR yet, but the diff is as follows.

Details

[ec2-user@ip-172-31-40-91 rootlesskit]$ gd
diff --git a/pkg/port/builtin/child/child.go b/pkg/port/builtin/child/child.go
index 83ab638..ebcdc67 100644
--- a/pkg/port/builtin/child/child.go
+++ b/pkg/port/builtin/child/child.go
@@ -122,7 +122,8 @@ func (d *childDriver) handleConnectRequest(c *net.UnixConn, req *msg.Request) er
        var dialer net.Dialer
        ip := req.IP
        if ip == "" {
-               ip = "127.0.0.1"
+               // ip = "127.0.0.1"
+               ip = req.ParentIP
        } else {
                p := net.ParseIP(ip)
                if p == nil {
diff --git a/pkg/port/builtin/msg/msg.go b/pkg/port/builtin/msg/msg.go
index f77468e..4b07161 100644
--- a/pkg/port/builtin/msg/msg.go
+++ b/pkg/port/builtin/msg/msg.go
@@ -19,10 +19,11 @@ const (

 // Request and Response are encoded as JSON with uint32le length header.
 type Request struct {
-       Type  string // "init" or "connect"
-       Proto string // "tcp", "tcp4", "tcp6", "udp", "udp4", "udp6"
-       IP    string
-       Port  int
+       Type     string // "init" or "connect"
+       Proto    string // "tcp", "tcp4", "tcp6", "udp", "udp4", "udp6"
+       IP       string
+       Port     int
+       ParentIP string
 }

 // Reply may contain FD as OOB
@@ -52,10 +53,11 @@ func Initiate(c *net.UnixConn) error {
 // that corresponds to the port spec.
 func ConnectToChild(c *net.UnixConn, spec port.Spec) (int, error) {
        req := Request{
-               Type:  RequestTypeConnect,
-               Proto: spec.Proto,
-               Port:  spec.ChildPort,
-               IP:    spec.ChildIP,
+               Type:     RequestTypeConnect,
+               Proto:    spec.Proto,
+               Port:     spec.ChildPort,
+               IP:       spec.ChildIP,
+               ParentIP: spec.ParentIP,
        }
        if _, err := lowlevelmsgutil.MarshalToWriter(c, &req); err != nil {
                return 0, err

[djdongjin]

I believe this is fixed with #3831, closing. Feel free to reopen if you still see the issue