No access to localhost ports #4

ivanov17 · 2020-10-29T10:34:52Z

As I previously wrote, I have been trying to use podman on CentOS 8 running on qemu/kvm virtual machine. I removed firewalld and I use nftables directly now.

Unfortunately, I have some troubles with my nftables rules and an issue related to access to localhost ports inside the virtual machine.

When I start nginx container and try to access to localhost on 80 port, the connection hangs until I interrupt it:

# curl -vvv localhost
* Rebuilt URL to: localhost/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.61.1
> Accept: */*
>

It also happens if I try to access 80 port using ip address of my virtual machine inside the VM:

# curl -vvv -4 192.168.122.101
* Rebuilt URL to: 192.168.122.101/
*   Trying 192.168.122.101...
* TCP_NODELAY set
* Connected to 192.168.122.101 (192.168.122.101) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.122.101
> User-Agent: curl/7.61.1
> Accept: */*
>

Outside the VM, I can access the container's port without any problem (If I remove the reject rule from the forward chain of the inet filter table, of course):

$ curl 192.168.122.101
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

When I stop nginx container, install package with another webserver on the virtual machine and then try to access 80 port on localhost inside the VM, I reach it successfully:

# curl localhost
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
  <title>Powered by lighttpd</title>
  <link rel="shortcut icon" href="favicon.ico" />
  <link rel="icon" href="favicon.ico" />
  <style type="text/css">
    <!--
      body { background: white; color: #666f85; text-align: center }
      img  { border: none }
    -->
  </style>
</head>
<body>

<p>
<img src="light_logo.png" width="249" height="239" alt="LIGHTTPD - fly light." />
</p>

<p>
<img src="poweredby.png" width="88" height="31" alt="Powered by GNU/Linux" />
<br />
<a href="http://www.lighttpd.net/"><img src="light_button.png" width="80" height="27" alt="Powered by Lighttpd" /></a>
</p>

</body>
</html>

I didn't create another CentOS 8 VM with default configuration running firewalld to test the access to localhost ports. But I use firewalld and podman in Fedora 32 on my home machine and I can access 80 port on localhost running the same nginx container.

It looks like there is an issue with rules set by CNI plugins.

This is a listing of my nftables ruleset when container is running:

# nft list ruleset
table inet filter {
	chain input {
		type filter hook input priority filter; policy accept;
		ct state established,related counter packets 86867 bytes 92621887 accept
		meta l4proto { icmp, ipv6-icmp } counter packets 4173 bytes 305844 accept
		iif "lo" counter packets 2746 bytes 285833 accept
		ip6 saddr fe80:: ip6 daddr fe80:: udp sport 547 udp dport 546 ct state new counter packets 0 bytes 0 accept
		tcp dport 22 ct state new counter packets 0 bytes 0 accept
		tcp dport 80 ct state new counter packets 15 bytes 900 accept
		tcp dport 443 ct state new counter packets 0 bytes 0 accept
		counter packets 1951 bytes 346572 reject with icmpx type admin-prohibited
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
	}

	chain output {
		type filter hook output priority filter; policy accept;
	}
}
table ip nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		jump cni-npo-50795b7a600c9b95c61de4b
	}

	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		jump cni-npr-50795b7a600c9b95c61de4b
	}

	chain output {
		type nat hook output priority -100; policy accept;
	}

	chain input {
		type nat hook input priority 100; policy accept;
	}

	chain cni-npr-50795b7a600c9b95c61de4b {
		iifname != "cni-podman0" tcp dport 80 dnat to 10.88.2.129:80
	}

	chain cni-npo-50795b7a600c9b95c61de4b {
		iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return
		iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 0 bytes 0 masquerade
	}
}
table ip raw {
	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip filter {
	chain forward {
		type filter hook forward priority filter; policy drop;
		jump cni-ffw-50795b7a600c9b95c61de4b
		oifname "cni-podman0" ip daddr 10.88.2.129 tcp dport 80 counter packets 3 bytes 180 accept
		log prefix "ip4 forward drop: "
		counter packets 0 bytes 0 drop
	}

	chain cni-ffw-50795b7a600c9b95c61de4b {
		oifname "cni-podman0" ip daddr 10.88.2.129 ct state established,related counter packets 30 bytes 3106 accept
		iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 27 bytes 4594 accept
		iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
	}
}
table ip6 nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		jump cni-npo-50795b7a600c9b95c61de4b
	}

	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		jump cni-npr-50795b7a600c9b95c61de4b
	}

	chain output {
		type nat hook output priority -100; policy accept;
	}

	chain input {
		type nat hook input priority 100; policy accept;
	}

	chain cni-npr-50795b7a600c9b95c61de4b {
		iifname != "cni-podman0" tcp dport 80 dnat to [fd10:88:7334:6d55::1e]:80
	}

	chain cni-npo-50795b7a600c9b95c61de4b {
		iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e ip6 daddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 return
	}
}
table ip6 raw {
	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip6 filter {
	chain forward {
		type filter hook forward priority filter; policy drop;
		jump cni-ffw-50795b7a600c9b95c61de4b
		oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e tcp dport 80 counter packets 0 bytes 0 accept
		log prefix "ip6 forward drop: "
		counter packets 0 bytes 0 drop
	}

	chain cni-ffw-50795b7a600c9b95c61de4b {
		oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e ct state established,related counter packets 0 bytes 0 accept
		iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 accept
		iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
	}
}

This is my CNI configuration:

# podman network inspect podman
[
	{
		"cniVersion": "0.4.0",
		"name": "podman",
		"plugins": [
			{
				"bridge": "cni-podman0",
				"hairpinMode": true,
				"ipMasq": false,
				"ipam": {
					"ranges": [
						[
							{
								"gateway": "10.88.0.1",
								"subnet": "10.88.0.0/16"
							}
						],
						[
							{
								"gateway": "fd10:88:7334:6d55::1",
								"subnet": "fd10:88:7334:6d55::/64"
							}
						]
					]
					"routes": [
						{
							"dst": "0.0.0.0/0"
						},
						{
							"dst": "::/0"
						}
					],
					"type": "host-local"
				},
				"isGateway": true,
				"type": "bridge"
			},
			{
				"capabilities": {
					"portMappings": true
				},
				"type": "cni-nftables-portmap"
			},
			{
				"forward_chain_name": "forward",
				"type": "cni-nftables-firewall"
			},
			{
				"type": "tuning"
			}
		]
	}
]

Podman version and system info:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

# podman system info
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: bc11c2e39bf61429aafb5d131e709a970036be24'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 1064951808
  MemTotal: 1915486208
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc10.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 1073737728
  SwapTotal: 1073737728
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: centos8.lan
  kernel: 4.18.0-193.19.1.el8_2.x86_64
  os: linux
  rootless: false
  uptime: 57h 7m 31.46s (Approximately 2.38 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
  - registry.access.redhat.com
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

The text was updated successfully, but these errors were encountered:

greenpau · 2020-10-30T15:31:44Z

@ivanov17 , can you access container IP address from host namespace?

greenpau · 2020-10-30T15:33:47Z

@ivanov17 , generally speaking ... if you try accessing localhost from host namespace hoping getting to container, I don't think it would work, because there are no rules accomplishing that. There is some work left "to do" with this plugin.

mebel-christ · 2021-03-08T06:16:50Z

+1
I am facing the same issue (for obvious reasons). This is kind of bugging me even so basically everything works fine it would be great to access the containers via localhost.
@greenpau do you plan to implement that?

greenpau · 2021-03-28T15:55:48Z

@micaeb , did not have a chance to look at it. Hoping someone else would fix it :-)

ghost · 2021-04-16T15:56:07Z

The problem is that prerouting hook will trigger only for packets from real device or bridge and won`t work for outgoing packages from local machine. For details see https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks. It is possible to do local DNAT as DNAT statements will work in output hook.

Don't tell nftables to block 127.0.0.1 from internal virtual networks. Just external physical networks. Signed-off-by: David Barr <dafydd@dafydd.com> - modified: README.md

greenpau · 2021-07-05T16:30:58Z

@ivanov17 , please check whether it work or not with the latest release. If it does, please close the issue.

ivanov17 mentioned this issue Oct 29, 2020

Unable to reject forwarding without rules set by plugins #3

Closed

ghost mentioned this issue Apr 16, 2021

Added jump rule to output chain for local DNAT. #7

Merged

ghost mentioned this issue May 9, 2021

Fixed localhost port mapping #11

Merged

dafydd2277 mentioned this issue Jul 5, 2021

Only jump to the port mapping chain if the traffic destination is the local system. #14

Merged

greenpau closed this as completed Nov 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No access to localhost ports #4

No access to localhost ports #4

ivanov17 commented Oct 29, 2020

greenpau commented Oct 30, 2020

greenpau commented Oct 30, 2020

mebel-christ commented Mar 8, 2021

greenpau commented Mar 28, 2021

ghost commented Apr 16, 2021 •

edited by ghost

Loading

greenpau commented Jul 5, 2021

No access to localhost ports #4

No access to localhost ports #4

Comments

ivanov17 commented Oct 29, 2020

greenpau commented Oct 30, 2020

greenpau commented Oct 30, 2020

mebel-christ commented Mar 8, 2021

greenpau commented Mar 28, 2021

ghost commented Apr 16, 2021 • edited by ghost Loading

greenpau commented Jul 5, 2021

ghost commented Apr 16, 2021 •

edited by ghost

Loading