WIP: firewall: add nftables backend #462
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -133,14 +133,218 @@ of the container as shown: | |
| - `-s 10.88.0.2 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT` | ||
| - `-d 10.88.0.2 -j ACCEPT` | ||
|
|
||
| ## nftables backend rule structure | ||
|
|
||
| The prerequisite for the backend is the existence of `filter` table and | ||
| the existence of `FORWARD` chain in the table. | ||
|
|
||
| A sample standalone config list (with the file extension `.conflist`) using | ||
| `nftables` backend might look like: | ||
|
|
||
| ```json | ||
| { | ||
| "cniVersion": "0.4.0", | ||
| "name": "podman", | ||
| "plugins": [ | ||
| { | ||
| "type": "bridge", | ||
| "bridge": "cni-podman0", | ||
| "isGateway": true, | ||
| "ipMasq": true, | ||
| "ipam": { | ||
| "type": "host-local", | ||
| "routes": [ | ||
| { | ||
| "dst": "0.0.0.0/0" | ||
| } | ||
| ], | ||
| "ranges": [ | ||
| [ | ||
| { | ||
| "subnet": "192.168.100.0/24", | ||
| "gateway": "192.168.100.1" | ||
| } | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "type": "portmap", | ||
| "capabilities": { | ||
| "portMappings": true | ||
| } | ||
| }, | ||
| { | ||
| "type": "firewall", | ||
| "backend": "nftables" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Prior to the invocation of CNI `firewall` plugin, the `FORWARD` chain in `filter` | ||
| table might be configured be as follows: | ||
|
|
||
| ``` | ||
| table ip filter { | ||
| chain FORWARD { # handle 1 | ||
| type filter hook forward priority filter; policy drop; | ||
| log prefix "IPv4 FORWARD drop: " flags all # handle 28 | ||
| counter packets 0 bytes 0 drop # handle 29 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Subsequently, the plugin creates "non-base chain", e.g. `cnins-3-4026543850-dummy0` | ||
| and link it to `FORWARD` chain | ||
| via [`jump` instruction](https://wiki.nftables.org/wiki-nftables/index.php/Jumping_to_chain). | ||
|
|
||
| ``` | ||
| table ip filter { | ||
| chain FORWARD { # handle 1 | ||
| type filter hook forward priority filter; policy drop; | ||
| jump cnins-3-4026543850-dummy0 # handle 10 | ||
| log prefix "IPv4 FORWARD drop: " flags all # handle 28 | ||
| counter packets 0 bytes 0 drop # handle 29 | ||
| } | ||
|
|
||
| chain cnins-3-4026543850-dummy0 { # handle 2 | ||
| oifname "dummy0" ip daddr 192.168.100.100 ct state established,related counter packets 0 bytes 0 accept # handle 3 | ||
| iifname "dummy0" ip saddr 192.168.100.100 counter packets 0 bytes 0 accept # handle 4 | ||
| iifname "dummy0" oifname "dummy0" counter packets 0 bytes 0 accept # handle 5 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The name of the chain is is prefixed with `CNINS-` and followed by `Dev` and `Ino` | ||
| of `Stat_t` struct. See [here](https://github.com/vishvananda/netns/blob/master/netns.go#L60) | ||
| for more information. | ||
|
|
||
| Generally, the testing of nftables backend of this plugin begins with defining | ||
| the data structure the plugin would receive when processing a request. | ||
| In this example, the plugin received single interface `dummy0`, with IPv4 and | ||
| IPv6 addresses. | ||
|
|
||
| ```json | ||
| { | ||
| "name": "test", | ||
| "type": "firewall", | ||
| "backend": "nftables", | ||
| "ifName": "dummy0", | ||
| "cniVersion": "0.4.0", | ||
| "prevResult": { | ||
| "interfaces": [ | ||
| { | ||
| "name": "dummy0" | ||
| } | ||
| ], | ||
| "ips": [ | ||
| { | ||
| "version": "4", | ||
| "address": "192.168.200.10/24", | ||
| "interface": 0 | ||
| }, | ||
| { | ||
| "version": "6", | ||
| "address": "2001:db8:1:2::1/64", | ||
| "interface": 0 | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Prior to running tests, the test harness does the following: | ||
|
|
||
| 1. creates `originalNS` namespace | ||
| 2. adds `dummy0` interface to `originalNS` via Netlink | ||
| 3. checks that the `dummy0` interface is available in the `originalNS` | ||
| 4. creates `targetNS` namespace | ||
|
|
||
| Upon the completion of the testing, the test harness does the following: | ||
|
|
||
| 1. closes `originalNS` namespace | ||
| 2. closes `targetNS` namespace | ||
|
|
||
| The tests in the harness start with `It()`. | ||
|
|
||
| Generally, a test contains a number of input arguments. In the case of | ||
| "installs nftables rules, checks the rules exist, then cleans up on delete using v4.0.x", | ||
| the test has the following arguments: | ||
|
|
||
| * container id: `dummy` | ||
| * the path to container namespace, i.e. `targetNS` | ||
| * the name of the interface | ||
| * the JSON payload containing a dummy request | ||
|
|
||
| The test uses the same arguments and runs the following operations in | ||
| `originalNS` namespace: | ||
|
|
||
| * `cmdAdd` | ||
| * `cmdCheck` | ||
| * `cmdDel` | ||
|
|
||
| The operations correspond to the following functions: | ||
|
|
||
| | **Operation** | **Function** | | ||
| | --- | --- | | ||
| | `cmdAdd` | `func (nb *nftBackend) Add(conf *FirewallNetConf, result *current.Result)` | | ||
| | `cmdCheck` | `func (nb *nftBackend) Del(conf *FirewallNetConf, result *current.Result)` | | ||
| | `cmdDel` | `func (nb *nftBackend) Check(conf *FirewallNetConf, result *current.Result)` | | ||
|
|
||
| The following command triggers the testing of `firewall` plugin: | ||
|
|
||
| ```bash | ||
| sudo go test -v ./plugins/meta/firewall | ||
| ``` | ||
|
|
||
| At the outset, the test outputs dummy host and container namespaces: | ||
|
|
||
| ``` | ||
| Host Namespace: /var/run/netns/cnitest-ba94096d-68e1-90c0-5e0a-4acf3a8339cd | ||
| Container Namespace: /var/run/netns/cnitest-762bc306-9af5-5882-af95-e011590ce8d3 | ||
| ``` | ||
|
|
||
| The knowing the last part of the namespace path helps inspecting namespaces | ||
| with `sudo ip netns exec` command. For example, the following command | ||
| show `nftables` tables, chains, and rules. | ||
|
|
||
| ```bash | ||
|
|
||
| $ sudo ip netns exec cnitest-ba94096d-68e1-90c0-5e0a-4acf3a8339cd nft list ruleset | ||
| table ip filter { | ||
| chain FORWARD { | ||
| type filter hook forward priority filter; policy drop; | ||
| jump cnins-3-4026550857-dummy0 | ||
| } | ||
|
|
||
| chain cnins-3-4026550857-dummy0 { | ||
| oifname "dummy0" ip daddr 192.168.100.100 ct state established,related counter packets 0 bytes 0 accept | ||
| iifname "dummy0" ip saddr 192.168.100.100 counter packets 0 bytes 0 accept | ||
| iifname "dummy0" oifname "dummy0" counter packets 0 bytes 0 accept | ||
| } | ||
| } | ||
| table ip6 filter { | ||
| chain FORWARD { | ||
| type filter hook forward priority filter; policy drop; | ||
| jump cnins-3-4026550857-dummy0 | ||
| } | ||
|
|
||
| chain cnins-3-4026550857-dummy0 { | ||
| oifname "dummy0" ip6 daddr 2001:db8:100:100::1 ct state established,related counter packets 0 bytes 0 accept | ||
| iifname "dummy0" ip6 saddr 2001:db8:100:100::1 counter packets 0 bytes 0 accept | ||
| iifname "dummy0" oifname "dummy0" counter packets 0 bytes 0 accept | ||
| } | ||
|
Comment on lines
+321
to
+337
There was a problem hiding this comment. Ideally these would be combined into one chain in a table of family There was a problem hiding this comment.
@erig0, it has a value in a sense that it is easier to delete the rule when you have a jump like this. It would be very hard to parse each rule. There was a problem hiding this comment. I get the jump to a dedicated chain. That 100% makes sense. I was saying I don't think you need to split the "ip" and "ip6" families. You could use "inet" which covers both IPv4 and IPv6. There was a problem hiding this comment.
@erig0, that I could do. 👍 It makes sense because IPv4/6 are all a part of the same construct. |
||
| } | ||
| ======= | ||
| The `CNI-FORWARD` chain first sends all traffic to `CNI-ADMIN` chain, which is intended as an user-controlled chain for custom rules that run prior to rules managed by the `firewall` plugin. The `firewall` plugin does not add, delete or modify rules in the `CNI-ADMIN` chain. | ||
|
|
||
| `CNI-FORWARD` chain: | ||
| - `-j CNI-ADMIN` | ||
|
|
||
| The chain name `CNI-ADMIN` can be overridden by specifying `iptablesAdminChainName` in the plugin configuration | ||
|
|
||
| ```json | ||
| { | ||
| "type": "firewall", | ||
| "backend": "iptables", | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not guaranteed to exist. It will likely exist on any machine that uses the
iptables-nftvariant of iptables as it implicitly create this table/chain. I would not assume the table/chain exists.CNI could use its own table and set of chains and thus avoid many interaction issues with
iptables-nft. However, a separate table and chains has its own issue related to netfilter hooks. The tl;dr is nftables allows multiple chains to use the same netfilter hook. A consequence of this is that packets that are accepted are still subject to the rules of other chains hooked into the hook type.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@erig0, I can create a chain. that is I do it when testing changes.
@dcbw , @mars1024 , agreed?