[Bug 1392] nft stalls on EAGAIN upon repeatedly flushing and populating a set

[bugzilla-daemon at netfilter.org]

https://bugzilla.netfilter.org/show_bug.cgi?id=1392

--- Comment #13 from kfm at plushkava.net ---
I have installed nftables (commit ca2e6e0) with libnftnl (commit a4db940) and
done some light testing with kernels 5.4.60 and 5.7.16. The good news is that
the exact issue reported by the opening comment no longer occurs. The bad news
is that there are some other issues that arise.

For both kernels, I conducted 9 tests amounting to the cross product of 3 set
configurations and 3 testing methodologies. The set configurations shall be
labelled with letters:

  A = type ipv4_addr; flags interval
  B = type ipv4_addr; flags interval; auto-merge
  C = type ipv4_addr; flags interval; auto-merge; timeout 4h5m

The testing methodologies shall be labelled with numbers:

  1 = (set empty; populate) x 1
  2 = (set populated; flush; populate) x 500
  3 = (set populated; no flush; populate) x 500

Hence, "1" means to attempt to populate the set just once after loading the
ruleset containing the set definition. "2" means to re-populate 500 times in
succession, with a flush command in-between. Likewise for "3", just with no
flush command between interations. In the case that there is a flush command,
it is integrated into the nft command stream, like so:-

  {
      if (( do_flush )); then
          echo 'flush set ip raw bogons'
      fi
      echo 'add element ip raw bogons { '
      grep -v '^#' /var/tmp/bogons.raw | tr '\n' ,
      echo ' }'
  } | nft -f - || exit

Below are the results of the nine tests, with wall time reported for some of
them.

  ╔════╦══════════════╦══════════════╗
  ║    ║    5.4.60    ║    5.7.16    ║
  ╠════╬══════════════╬══════════════╣
  ║ A1 ║      OK      ║      OK      ║
  ╠════╬══════════════╬══════════════╣
  ║ A2 ║ OK (14.015s) ║ OK (9.355s)  ║
  ╠════╬══════════════╬══════════════╣
  ║ A3 ║ OK (47.325s) ║ OK (30.718s) ║
  ╠════╬══════════════╬══════════════╣
  ║ B1 ║      OK      ║      OK      ║
  ╠════╬══════════════╬══════════════╣
  ║ B2 ║ OK (13.274s) ║ OK (9.934s)  ║
  ╠════╬══════════════╬══════════════╣
  ║ B3 ║     FAIL     ║     FAIL     ║
  ╠════╬══════════════╬══════════════╣
  ║ C1 ║      OK      ║      OK      ║
  ╠════╬══════════════╬══════════════╣
  ║ C2 ║ OK (13.514s) ║ OK (8.941s)  ║
  ╠════╬══════════════╬══════════════╣
  ║ C3 ║     FAIL     ║     FAIL     ║
  ╚════╩══════════════╩══════════════╝

Let's begin with the good. There is no apparent difference in the behaviour of
either kernel except that 5.7 is faster. Tests {A,B,C}2 pass, implying that I
am now able to successfuly use the flush-then-add method of atomically
repopulating a set.

Now for the bad. Test A3 passes, even though the auto-merge flag isn't defined.
On the other hand, tests {B,C}3 - where the auto-merge flag IS defined - fail.
This is confusing to me. If anything, I would expect A3 to fail because all of
the elements being added already exist. Here are the error messages reported in
the cases of failure:

  /dev/stdin:2:820-833: Error: interval overlaps with an existing one
  /dev/stdin:1:1-20343: Error: Could not process rule: File exists

Finally, while test A3 passes, it is noticeably slower than test A2, which
incorporates a flush command for each iteration. I arranged for two similar
tests, using a hash:net ipset. One test relied on the use of "ipset flush" for
each iteration, and the other relied on the use "ipset -exist". I understand
that it might not be entirely fair to compare the -exist option to the
auto-merge flag. Nevertheless, I found that both of the ipset test cases
completed in approximately 18 seconds, implying that the -exist option has a
low performance impact.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200823/a77d4c5e/attachment.html>