[Bug 1439] New: Atomically updating/reloading a large set with nft -f is excessively slow

[bugzilla-daemon at netfilter.org]

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

            Bug ID: 1439
           Summary: Atomically updating/reloading a large set with nft -f
                    is excessively slow
           Product: nftables
           Version: unspecified
          Hardware: x86_64
                OS: Debian GNU/Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: public_timo.s at silentcreek.de

Hi,

I'm trying to move my iptables/ipset setups to nftables. Some of these setups
involve fairly large sets and updating those atomically does not seem to be
practical due to the excessive amount of time it takes to reload the set via a
script file.

Assume a set that was initialized like this:
  nft add set inet filter ipv6_bogons { type ipv6_addr; flags interval; }

And then load the attached example ipv6_bogons.nft with `nft -f' to populate
the set with elements. When you first load that file, it's a matter of seconds
(or less). But if you try to load the set another time in order to reload it
(the script contains a flush set statement at the top), it takes excessively
long.

For reference: On an 8th generation Intel Core i7 with a turbo frequency of
5GHz, reloading the file takes over 3 minutes(!) as opposed to ~0.6s when
loaded initially. If you do this on less powerful hardware, such as a dual-core
1GHz ARMv7 SoC, first loading the file takes a few seconds, but the second time
seems to take forever - I aborted the experiment after 45 minutes! These
loading times are just not practical, even on powerful hardware, when you're
working with multiple large sets and looking for a way to reload them
atomically. (Obviously, you wouldn't want to reload the exact same set, but
after updating the file with added or deleted elements.)

The only workaround so far that I found was to reload the complete ruleset
(with  an include statement that references the attached ipv6_bogons.nft file).
This works quicky, but has a few drawbacks, naturally, such as resetting
counters and losing dynamically populated sets (with elements that are not
permanently saved to nft script files).

I have tested this both with nftables 0.9.0-2 (Debian 10) as well as 0.9.3-2
(Ubuntu 20.04).

If there is anything I can do to speed this up and reload a large set
atomically, please let me know.

Cheers,

Timo

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200701/bd639c52/attachment.html>