libnetfilter_queue vs libipq

[Hong Yan]

Here's the experiment setup: we connected 3 emulab (www.emulab.net) Linux
(kernel 2.6.12) PCs (3GHZ CPU and 2G MB RAM) via 100M bps Ethernet
interfaces to form a client-firewall-server chain, we run a TCP/UDP
measurement tool called iPerf on the client and the server, when the
iptables are clean, the TCP throughput is 95M bps. We then run the following
libipq test:

yum install iptables-devel 
/sbin/iptables -F FORWARD
/sbin/iptables -A FORWARD -p tcp --dport 5001 -j QUEUE
/sbin/modprobe iptable_filter
/sbin/modprobe ip_queue
./ipq-test

And the throughput dropped to 2M bps. I have also tried snort_inline and the
performance is as bad. Did I do something wrong?

Here's the ipq-test source code (also attached):

#include <linux/netfilter.h>
#include <libipq.h>
#include <stdio.h>

#define BUFSIZE 2048 

static void die(struct ipq_handle *h)
{
        ipq_perror("passer");
        ipq_destroy_handle(h);
        exit(1);
}

int main(int argc, char **argv)
{
        int status;
        unsigned char buf[BUFSIZE];


        struct ipq_handle *h;
        
        h = ipq_create_handle(0, PF_INET);
        if (!h)
                die(h);
                
        status = ipq_set_mode(h, IPQ_COPY_PACKET, BUFSIZE);
        if (status < 0)
                die(h);
                
        do{

                status = ipq_read(h, buf, BUFSIZE, 0);
                if (status < 0)
                        die(h);
                        
                switch (ipq_message_type(buf)) {
                        case NLMSG_ERROR:
                                fprintf(stderr, "Received error message
%d\n",
                                        ipq_get_msgerr(buf));
                                break;
                                


                        case IPQM_PACKET: {
                                ipq_packet_msg_t *m = ipq_get_packet(buf);
                                status = ipq_set_verdict(h, m->packet_id,
                                                         NF_ACCEPT, 0,
NULL);
                                if (status < 0)
                                        die(h);
                                break;
                        }
                        
                        default:
                                fprintf(stderr, "Unknown message type!\n");
                                break;
                }
        } while (1);
        

        ipq_destroy_handle(h);
        return 0;
}



-----Original Message-----
From: Harald Welte [mailto:laforge at gnumonks.org] 
Sent: Thursday, January 12, 2006 4:55 AM
To: Hong Yan
Cc: netfilter-devel at lists.netfilter.org
Subject: Re: libnetfilter_queue vs libipq

On Wed, Jan 11, 2006 at 09:44:33PM -0500, Hong Yan wrote:
 
> I'm a graduate student at Carnegie Mellon University. We built a firewall
> using libipq and later switched to libnetfilter_queue because we found
that
> libnetfilter_queue significantly outperforms libipq. A simple testing
> program using libnetfilter_queue to intercept and then accept all packets
> has 94.2 M bps TCP throughput but with libipq it can only achieve 2M bps.

I'm very surprised by your results.  Maybe you can describe the
application in more detail (as in how the queue mechanism is used,
whether you reinject or only send a verdict, if you process one packet
at a time, how long your queue lengths get, what kind of hardware, is
the app multithreaded, etc.)

> We wonder what is causing the big difference in performance. Could you
shed
> us some light on this?

Other real-world application users of libipq / libnetfilter_queue have
show an almost equal performance for both systems.

In theory, libnetfilter_queue should even have a little bit more
overhead then libipq (at the gain of more flexibility).  

Therefore, I think it is important for us to learn, what exactly causes
this performance difference in your case.

Please follow-up to netfilter-devel at lists.netfilter.org, since there
certainly are other developers who would like to learn about this issue.

Thanks!

-- 
- Harald Welte <laforge at gnumonks.org>          	        http://gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipq-test.c
Type: application/octet-stream
Size: 2122 bytes
Desc: not available
Url : /pipermail/netfilter-devel/attachments/20060112/3fcb9374/ipq-test-0001.obj