fix: allow to propagate the address specified in -p option #477

haytok · 2025-01-03T18:12:20Z

When containerd is running as a root user and an address other than 127.0.0.1, such as 127.0.0.2, is specified in -p when nerdctl run is used to start a container, the combination of the address and port number allows connections to an application running in the container.

Specifically, curl 127.0.0.2:8080 will access a nginx application running on port 80 in the container created by the following command.

$ sudo nerdctl run --name nginx -d -p 127.0.0.2:8080:80 nginx
7abce48d42ef809613365ffaa54d9526e8388c78f5f5c2d8a2850628543ca27e

$ sudo nerdctl ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED          STATUS    PORTS                     NAMES
7abce48d42ef    docker.io/library/nginx:latest    "/docker-entrypoint.…"    3 seconds ago    Up        127.0.0.2:8080->80/tcp    nginx

$ curl 127.0.0.2:8080
<!DOCTYPE html>
...

On the other hand, when running containerd as a non-root user, similarly, suppose we start a container with an address other than 127.0.0.1, such as 127.0.0.2 for -p.
In this case, curl 127.0.0.2:8080 will not connect to a nginx application running inside the container.

Details are described below.

$ nerdctl run --name nginx -d -p 127.0.0.2:8080:80 nginx
7687f9612afe6847fb3946254718fafef935ffdc05f9cbb4496fc40dd35a6abc

$ nerdctl ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED          STATUS    PORTS                     NAMES
7687f9612afe    docker.io/library/nginx:latest    "/docker-entrypoint.…"    5 seconds ago    Up        127.0.0.2:8080->80/tcp    nginx

$ curl 127.0.0.2:8080
curl: (56) Recv failure: Connection reset by peer

When a connection is requested from the outside to the container in Parent NetNS, the rootlesskit port driver relays the connection from Parent NetNS to Child NetNS.
When, for example, an address such as 127.0.0.2 is specified in -p, Child NetNS need to establish a connection to the specified address.

However, the current implementation will establish a connection to 127.0.0.1 even when 127.0.0.2 is specified in -p.

The behavior of not propagating the address specified by -p is assumed at this time, as described in the following document.

https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md#port-drivers

--port-driver Throughput Source IP
...
builtin 30.0 Gbps Always 127.0.0.1
...
The builtin driver is fast, but be aware that the source IP is not propagated and always set to 127.0.0.1.

However, an issue to improve this behavior is reported below.

Port forwarding doesn't work for non-127.0.0.1 localhost containerd/nerdctl#3539

Therefore, this commit fixes when running containerd as a non-root user, the address specified in -p will be propagated.

When containerd is running as a root user and an address other than `127.0.0.1`, such as `127.0.0.2`, is specified in `-p` when `nerdctl run` is used to start a container, the combination of the address and port number allows connections to an application running in the container. Specifically, `curl 127.0.0.2:8080` will access a nginx application running on port `80` in the container created by the following command. ``` $ sudo nerdctl run --name nginx -d -p 127.0.0.2:8080:80 nginx 7abce48d42ef809613365ffaa54d9526e8388c78f5f5c2d8a2850628543ca27e $ sudo nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7abce48d42ef docker.io/library/nginx:latest "/docker-entrypoint.…" 3 seconds ago Up 127.0.0.2:8080->80/tcp nginx $ curl 127.0.0.2:8080 <!DOCTYPE html> ... ``` On the other hand, when running containerd as a non-root user, similarly, suppose we start a container with an address other than `127.0.0.1`, such as `127.0.0.2` for -p. In this case, curl `127.0.0.2:8080` will not connect to a nginx application running inside the container. Details are described below. ``` $ nerdctl run --name nginx -d -p 127.0.0.2:8080:80 nginx 7687f9612afe6847fb3946254718fafef935ffdc05f9cbb4496fc40dd35a6abc $ nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7687f9612afe docker.io/library/nginx:latest "/docker-entrypoint.…" 5 seconds ago Up 127.0.0.2:8080->80/tcp nginx $ curl 127.0.0.2:8080 curl: (56) Recv failure: Connection reset by peer ``` When a connection is requested from the outside to the container in Parent NetNS, the rootlesskit port driver relays the connection from Parent NetNS to Child NetNS. When, for example, an address such as `127.0.0.2` is specified in -p, Child NetNS need to establish a connection to the specified address. However, the current implementation will establish a connection to `127.0.0.1` even when `127.0.0.2` is specified in -p. The behavior of not propagating the address specified by -p is assumed at this time, as described in the following document. - https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md#port-drivers > --port-driver Throughput Source IP > ... > builtin 30.0 Gbps Always 127.0.0.1 > ... > The builtin driver is fast, but be aware that the source IP is not propagated and always set to 127.0.0.1. However, an issue to improve this behavior is reported below. - containerd/nerdctl#3539 Therefore, this commit fixes when running containerd as a non-root user, the address specified in -p will be propagated. Signed-off-by: Hayato Kiwata <haytok@amazon.co.jp>

AkihiroSuda · 2025-01-07T17:08:43Z

Thanks, can we have a test?

rootlesskit/.github/workflows/main.yaml

Lines 40 to 54 in 9c9049a

    
           - name: "Integration test: exit-code" 
        
             run: docker run --rm --privileged rootlesskit:test-integration ./integration-exit-code.sh 
        
           - name: "Integration test: propagation" 
        
             run: docker run --rm --privileged rootlesskit:test-integration ./integration-propagation.sh 
        
           - name: "Integration test: propagation (with `mount --make-rshared /`)" 
        
             run: docker run --rm --privileged rootlesskit:test-integration sh -exc "sudo mount --make-rshared / && ./integration-propagation.sh" 
        
           - name: "Integration test: restart" 
        
             run: docker run --rm --privileged rootlesskit:test-integration ./integration-restart.sh 
        
           - name: "Integration test: port" 
        
             # NOTE: "--net=host" is a bad hack to enable IPv6 
        
             run: docker run --rm --net=host --privileged rootlesskit:test-integration ./integration-port.sh 
        
           - name: "Integration test: IPv6 routing" 
        
             run: docker run  --rm --privileged --sysctl net.ipv6.conf.all.disable_ipv6=0 rootlesskit:test-integration ./integration-ipv6.sh 
        
           - name: "Integration test: systemd socket activation" 
        
             run: docker run --rm --net=none --privileged rootlesskit:test-integration ./integration-systemd-socket.sh

haytok · 2025-01-08T14:20:08Z

Thanks for the comment!!!

My Investigation

I've checked the contents of the integration test in main.yaml such as integration-port.sh, which is run by specifying --port-driver=builtin.
However, even when we add tests to these, we do not expect to be able to test for this nerdctl issue (containerd/nerdctl#3539).

At first I thought we could add the following test to integration-port.sh.

test_port builtin http://127.0.0.2:8080 "should success" -p 127.0.0.2:8080:80/tcp

However, I have confirmed in my development environment that this test case passes without applying this PR fix.

Details

ubuntu@ip-172-31-33-251:~/rootlesskit$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	diff.patch
	hack/issue_3539.sh

nothing added to commit but untracked files present (use "git add" to track)

# Run this integration test
ubuntu@ip-172-31-33-251:~/rootlesskit$ ls hack/issue_3539.sh
hack/issue_3539.sh

ubuntu@ip-172-31-33-251:~/rootlesskit$ cat hack/issue_3539.sh
#!/bin/bash
source $(realpath $(dirname $0))/common.inc.sh

# test_port PORT_DRIVER CURL_URL EXPECTATION [ROOTLESSKIT ARGS...]
function test_port() {
	args="$@"
	port_driver="$1"
	curl_url="$2"
	expectation="$3"
	shift
	shift
	shift
	rootlesskit_args="$@"
	INFO "Testing port_driver=\"${port_driver}\" curl_url=\"${curl_url}\" expectation=\"${expectation}\" rootlesskit_args=\"${rootlesskit_args}\""
	tmp=$(mktemp -d)
	state_dir=${tmp}/state
	html_dir=${tmp}/html
	mkdir -p ${html_dir}
	echo "test_port ($args)" >${html_dir}/index.html
	$ROOTLESSKIT \
		--state-dir=${state_dir} \
		--net=slirp4netns \
		--disable-host-loopback \
		--copy-up=/etc \
		--port-driver=${port_driver} \
		${rootlesskit_args} \
		busybox httpd -f -v -p 80 -h ${html_dir} \
		2>&1 &
	pid=$!
	sleep 1

	set +e
	curl -fsSL ${curl_url}
	code=$?
	set -e
	if [ "${expectation}" = "should success" ]; then
		if [ ${code} != 0 ]; then
			ERROR "curl exited with ${code}"
			exit ${code}
		fi
	elif [ "${expectation}" = "should fail" ]; then
		if [ ${code} = 0 ]; then
			ERROR "curl should not success"
			exit 1
		fi
	else
		ERROR "internal error"
		exit 1
	fi

	INFO "Test pasing, stopping httpd (\"exit status 255\" is negligible here)"
	kill -SIGTERM $(cat ${state_dir}/child_pid)
	wait $pid >/dev/null 2>&1 || true
	rm -rf $tmp
}

INFO "===== Port driver: builtin ====="
INFO "=== protocol \"tcp\" listens on both v4 and v6 ==="
test_port builtin http://127.0.0.2:8080 "should success" -p 127.0.0.2:8080:80/tcp

INFO "===== PASSING ====="

# Build rootlesskit and install 
ubuntu@ip-172-31-33-251:~/rootlesskit$ make && sudo make install
...

# Build docker image for integration test
ubuntu@ip-172-31-33-251:~/rootlesskit$ DOCKER_BUILDKIT=1 sudo docker build -t rootlesskit:test-integration --target test-integration .
...

# Run the integration test
ubuntu@ip-172-31-33-251:~/rootlesskit$ sudo docker run --rm --privileged rootlesskit:test-integration ./issue_3539.sh
[INFO] ===== Port driver: builtin =====
[INFO] === protocol "tcp" listens on both v4 and v6 ===
[INFO] Testing port_driver="builtin" curl_url="http://127.0.0.2:8080" expectation="should success" rootlesskit_args="-p 127.0.0.2:8080:80/tcp"
[rootlesskit:parent] error: no command specified

In the first place, this modification allows curl 127.0.0.2:8080:80 to a container started with nerdctl run --name nginx -d -p 127.0.0.2:8080:80 nginx, which is an improvement over nerdctl and containerd (rootless mode).
So, I think it is difficult to implement tests for this fix on rootlesskit side ...

Question

On the other hand, I think we need to add the following test on the nerdctl side for this fix, what do you think ???
I would be appreciated if you could give me some advice.

Details

[ec2-user@ip-172-31-40-91 nerdctl]$ git diff cmd/nerdctl/
diff --git a/cmd/nerdctl/container/container_run_linux_test.go b/cmd/nerdctl/container/container_run_linux_test.go
index dc33702e..357ca1cf 100644
--- a/cmd/nerdctl/container/container_run_linux_test.go
+++ b/cmd/nerdctl/container/container_run_linux_test.go
@@ -567,3 +567,42 @@ func TestIssue3568(t *testing.T) {

        testCase.Run(t)
 }
+
+// TestIssue3568 tests https://github.com/containerd/nerdctl/issues/3539
+func TestIssue3539(t *testing.T) {
+       testCase := nerdtest.Setup()
+
+       const (
+               host     = "127.0.0.2"
+               hostPort = 8080
+       )
+       address := fmt.Sprintf("%s:%d", host, hostPort)
+
+       testCase.SubTests = []*test.Case{
+               {
+                       Description: "Issue #3539 - Access to a container running when 127.0.0.2 is specified in -p in rootless mode.",
+                       Setup: func(data test.Data, helpers test.Helpers) {
+                               helpers.Ensure("run", "-d", "--name", data.Identifier(), "-p", fmt.Sprintf("%s:80", address), testutil.NginxAlpineImage)
+                       },
+                       Command: func(data test.Data, helpers test.Helpers) test.TestableCommand {
+                               return helpers.Custom("curl", "-s", address)
+                       },
+                       Cleanup: func(data test.Data, helpers test.Helpers) {
+                               helpers.Anyhow("rm", "-f", data.Identifier())
+                       },
+                       Expected: func(data test.Data, helpers test.Helpers) *test.Expected {
+                               return &test.Expected{
+                                       ExitCode: 0,
+                                       Errors:   []error{},
+                                       Output: test.All(
+                                               func(stdout string, info string, t *testing.T) {
+                                                       assert.Assert(t, strings.Contains(string(stdout), testutil.NginxAlpineIndexHTMLSnippet))
+                                               },
+                                       ),
+                               }
+                       },
+               },
+       }
+
+       testCase.Run(t)
+}

Note that I've confirmed that this test in nerdctl succeeds after applying the fix for rootkesskit.

Details

# applied this fix to rootlesskit
[ec2-user@ip-172-31-40-91 rootlesskit]$ git shortlog -p -1
Hayato Kiwata (1):
      fix: allow to propagate the address specified in -p option

# build and install rootlesskit
[ec2-user@ip-172-31-40-91 rootlesskit]$ make && sudo make install
...

# restart containerd
[ec2-user@ip-172-31-40-91 rootlesskit]$ sudo systemctl restart user@1000.service

[ec2-user@ip-172-31-40-91 rootlesskit]$ cd ../nerdctl/

[ec2-user@ip-172-31-40-91 nerdctl]$ go test -p 1 ./cmd/nerdctl/... -run "TestIssue3539.*" -count=1
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.011s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/apparmor	0.012s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/builder	0.010s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/completion	0.010s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/compose	0.012s [no tests to run]
# This is the result of passing the above test (TestIssue3539)
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/container	1.247s
?   	github.com/containerd/nerdctl/v2/cmd/nerdctl/helpers	[no test files]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/image	0.011s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/inspect	0.011s [no tests to run]
?   	github.com/containerd/nerdctl/v2/cmd/nerdctl/internal	[no test files]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs	0.010s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/issues	0.011s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/login	0.016s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/namespace	0.015s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	0.011s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/system	0.012s [no tests to run]
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/volume	0.011s [no tests to run]

AkihiroSuda

Thanks

…in rootless mode When running a rootless container, specifying an address such as 127.0.0.2 for the -p option, we will not be able to access the published container through that address. This behavior is reported in the following issue: - containerd#3539 This behavior is caused by the behavior in rootlesskit, and the following pull request has been made to improve the behavior. - rootless-containers/rootlesskit#477 Therefore, this commit adds a test to ensure that the behavior of the issue has been improved. Signed-off-by: Hayato Kiwata <haytok@amazon.co.jp>

AkihiroSuda added this to the v2.3.2 milestone Jan 9, 2025

AkihiroSuda approved these changes Jan 9, 2025

View reviewed changes

AkihiroSuda merged commit f669d45 into rootless-containers:master Jan 9, 2025
5 checks passed

haytok deleted the nerdctl_issue_3539 branch January 10, 2025 00:55

haytok mentioned this pull request Jan 22, 2025

Fix port forwarding not working for non-127.0.0.1 localhost in rootless containerd/nerdctl#3831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: allow to propagate the address specified in -p option #477

fix: allow to propagate the address specified in -p option #477

haytok commented Jan 3, 2025

AkihiroSuda commented Jan 7, 2025

haytok commented Jan 8, 2025 •

edited

Loading

AkihiroSuda left a comment

fix: allow to propagate the address specified in -p option #477

fix: allow to propagate the address specified in -p option #477

Conversation

haytok commented Jan 3, 2025

AkihiroSuda commented Jan 7, 2025

haytok commented Jan 8, 2025 • edited Loading

My Investigation

Question

AkihiroSuda left a comment

Choose a reason for hiding this comment

haytok commented Jan 8, 2025 •

edited

Loading