[Bug 1735] New: Adding nftables interval sets progressively gets slower and makes the nft CLI less responsive with each added set

[bugzilla-daemon at netfilter.org]

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

            Bug ID: 1735
           Summary: Adding nftables interval sets progressively gets
                    slower and makes the nft CLI less responsive with each
                    added set
           Product: nftables
           Version: 1.0.x
          Hardware: All
                OS: All
            Status: NEW
          Severity: major
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: anton.khazan at gmail.com

Created attachment 734
  --> https://bugzilla.netfilter.org/attachment.cgi?id=734&action=edit
shell script demonstrating the bug

The title basically describes the issue.

Steps to reproduce the issue:
Take an ip list for any country (in most of my tests I've been using the ip
list for GB fetched from ipdeny, which currently has 8589 ipv4 ip blocks),
measure time taking to load it into a new set, measure time adding the same ip
list again in a new set under a new name (without removing the 1st one), repeat
n times. Each new iteration takes a little longer. On the OpenWRT installation
in VM, 1st iteration takes 0.12s. 10th iteration takes 0.47s. 40th iteration
takes 1.59s.

Getting a response from the command 'nft list tables' also takes progressively
longer with each added set, as well as any other nft CLI operation involving
the table under test. Starting with 0.04s after 1st iteration and up to 1.49s
after the 40th iteration.

Monitoring memory use while running this test also reveals that the transient
memory use spikes (similar to those reported in Bug 1584) get progressively
larger as you go, reaching about 180MB around the 40th iteration (i've been
just monitoring memory use via the 'top' utility which samples once a second at
most, so I might have missed the peak of the spike).

Using the same ip list is not an essential part of the bug, as well as the
specific list for GB. I picked GB because it's a mid-sized list, and re-using
it because this way it is possible to get comparable results. The issue also
extends to ipv6 ip lists.

nft CLI operations involving other tables seem unaffected.

This is a pretty big problem for applications like geoip, especially so on
embedded devices like routers with limited memory.

I'm not sure whether this belongs to the kernel component or nft userspace.
Filing this under nft userspace component in the meanwhile.

Hardware used for testing: Intel i7-4770
Same behavior on Linux Mint (kernel 6.2.0, nftables 1.0.2) and OpenWRT x86-64
installed in a VM (kernel 5.15.137, nftables 1.0.8).

A shell script reproducing the issue is attached.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240130/f11cf698/attachment-0001.html>