Unable to reject forwarding without rules set by plugins

[ivanov17]

Hello!

I have been trying to use podman on CentOS 8 running on qemu/kvm virtual machine. I removed firewalld and I use nftables directly now.

Thank you for your plugins, they work very well in general, but I have some troubles with my nftables rules and an issue regarding access to localhost ports on the VM.

For testing purposes I use simple ruleset containing inet filter table:

table inet filter {
        chain input {
                type filter hook input priority filter; policy accept;
                ct state established,related accept
                meta l4proto { icmp, ipv6-icmp } accept
                iif "lo" accept 
                ip6 saddr fe80:: ip6 daddr fe80:: udp sport 547 udp dport 546 ct state new accept
                tcp dport 22 ct state new accept
                tcp dport 80 ct state new accept
                tcp dport 443 ct state new accept
                reject with icmpx type admin-prohibited
        }
        chain forward {
                type filter hook forward priority filter; policy accept;
                reject with icmpx type admin-prohibited
        }
        chain output {
                type filter hook output priority filter; policy accept;
        }
}

But when I run pod or container using podman the collision with rules set by plugins appears, because forward chains in ip filter and ip6 filter tables are placed after inet filter table:

table ip nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		jump cni-npo-50795b7a600c9b95c61de4b
	}
	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		jump cni-npr-50795b7a600c9b95c61de4b
	}
	chain output {
		type nat hook output priority -100; policy accept;
	}
	chain input {
		type nat hook input priority 100; policy accept;
	}
	chain cni-npr-50795b7a600c9b95c61de4b {
		iifname != "cni-podman0" tcp dport 80 dnat to 10.88.2.129:80
	}
	chain cni-npo-50795b7a600c9b95c61de4b {
		iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return
		iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 0 bytes 0 masquerade
	}
}
table ip raw {
	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip filter {
	chain forward {
		type filter hook forward priority filter; policy drop;
		jump cni-ffw-50795b7a600c9b95c61de4b
		oifname "cni-podman0" ip daddr 10.88.2.129 tcp dport 80 counter packets 2 bytes 120 accept
		log prefix "ip4 forward drop: "
		counter packets 0 bytes 0 drop
	}
	chain cni-ffw-50795b7a600c9b95c61de4b {
		oifname "cni-podman0" ip daddr 10.88.2.129 ct state established,related counter packets 24 bytes 2715 accept
		iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 22 bytes 3476 accept
		iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
	}
}
table ip6 nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		jump cni-npo-50795b7a600c9b95c61de4b
	}
	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		jump cni-npr-50795b7a600c9b95c61de4b
	}
	chain output {
		type nat hook output priority -100; policy accept;
	}
	chain input {
		type nat hook input priority 100; policy accept;
	}
	chain cni-npr-50795b7a600c9b95c61de4b {
		iifname != "cni-podman0" tcp dport 80 dnat to [fd10:88:7334:6d55::1e]:80
	}
	chain cni-npo-50795b7a600c9b95c61de4b {
		iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e ip6 daddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 return
	}
}
table ip6 raw {
	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip6 filter {
	chain forward {
		type filter hook forward priority filter; policy drop;
		jump cni-ffw-50795b7a600c9b95c61de4b
		oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e tcp dport 80 counter packets 0 bytes 0 accept
		log prefix "ip6 forward drop: "
		counter packets 0 bytes 0 drop
	}
	chain cni-ffw-50795b7a600c9b95c61de4b {
		oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e ct state established,related counter packets 0 bytes 0 accept
		iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 accept
		iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
	}
}

If I run the nginx container with podman run -dt --name nginx-test --rm --publish 80:80 docker.io/library/nginx:stable, I'm unable to connect to the container's 80 port using ip address of the virtual machine.

When I remove the reject rule from the forward chain of the inet filter table, chains in ip filter and ip6 filter tables work as expected. In that case if I run nginx container, I have a connection to the container and I can see nginx default page.

But in that case I have to always have a running pod or container, because the ruleset doesn't have any reject rule regarding forwarding if plugin rules are not applied.

I confused a bit that my ruleset needs changes to work with podman and don't work properly if I don't start any container. I think I should to change the priority of my forward chain for applying it after the plugins' forward chains. But maybe there is another solution?

It seems that applying of plugins rules without manual rewriting existing rules would be a better solution.

This is my CNI configuration:

# podman network inspect podman
[
	{
		"cniVersion": "0.4.0",
		"name": "podman",
		"plugins": [
			{
				"bridge": "cni-podman0",
				"hairpinMode": true,
				"ipMasq": false,
				"ipam": {
					"ranges": [
						[
							{
								"gateway": "10.88.0.1",
								"subnet": "10.88.0.0/16"
							}
						],
						[
							{
								"gateway": "fd10:88:7334:6d55::1",
								"subnet": "fd10:88:7334:6d55::/64"
							}
						]
					]
					"routes": [
						{
							"dst": "0.0.0.0/0"
						},
						{
							"dst": "::/0"
						}
					],
					"type": "host-local"
				},
				"isGateway": true,
				"type": "bridge"
			},
			{
				"capabilities": {
					"portMappings": true
				},
				"type": "cni-nftables-portmap"
			},
			{
				"forward_chain_name": "forward",
				"type": "cni-nftables-firewall"
			},
			{
				"type": "tuning"
			}
		]
	}
]

Podman version and system info:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

# podman system info
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: bc11c2e39bf61429aafb5d131e709a970036be24'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 1064951808
  MemTotal: 1915486208
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc10.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 1073737728
  SwapTotal: 1073737728
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: centos8.lan
  kernel: 4.18.0-193.19.1.el8_2.x86_64
  os: linux
  rootless: false
  uptime: 57h 7m 31.46s (Approximately 2.38 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
  - registry.access.redhat.com
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

[greenpau]

@ivanov17 , there is a lot to digest here. Will look at it tomorrow. Also, could you please split this into multiple issue? It will be easier to track this.

[greenpau]

@ivanov17 , when I was designing the plugin, I did not want to use inet table. I did not account for inet table.

[SilverBut]

hi @greenpau. I think currently inet table is still not being used. Do you think there might be any difficulties to implement it? I'm not sure if ip/ip6 table could work well with inet table, since it does not work in my environment...Some quick debug shows that it the incoming UDP packet is successfully dnated, but no future logs reported. Change ip to inet (and modify some rules manually) also does not work.

P.S. Seems the PR in containernetworking/plugins#462 have been silent for a while. I think they might not going to merge it...since the PR even lived during several COVID variants.