Possible bug in conntrack

Martin Rosenmüller martin.rosenmueller at gmx.de
Thu Jul 19 09:21:04 CEST 2007


Hi all,

we are planning to build a loadsharing cluster of nodes based on
iptables CLUSTERIP target.

In our tests some real strange things happended.
For simplicity i will explain the problem with only one active node in
the cluster.
My configuration on the cluster node was:

	- eth0 had 172.16.29.121 (real ip of the node)
	- eth0:1 had 172.16.29.200 (virtual cluster ip)
	- the clustered service was tcp port 80 (http) (other ports didn't work
also)
	- please find my firewall-configuration script attached
	- the firewall works as follows:
		- first the stateful rule match ctstate in state	 ESTABLISHED, RELATED
		- then the stateful rule match ctstate in state INVALID
		- afterwards the CLUSTERIP target rule
		- followed by the rule to accept new connections on the virtual
cluster ip on port 80
		- finally the clean up rule to drop all traffic that didn't match up
to now

Surely i didn't forget to enter "echo +2
> /proc/net/ipt_CLUSTERIP/172.16.29.200" to accept all cluster traffic
on the only active node, the webserver was running and bind to the real
and the virtual cluster ip.

If you than connect to the cluster on the virtual ip 172.16.29.200 from
a client node on the same net (client has ip 172.16.29.211), you will
almost get no answer (client.pcap). The cluster node seems not to accept
any traffic than the first syn packet, although it answers with syn,ack.
Please see the network dump attached (clusternode.pcap). We have checked
the multicast mac, which seems to be ok. Logging in messages (messages)
is OK, too. All packets arriving the cluster node after the syn packet
are accepted by conntrack in state ESTABLISHED, which also seems to be
OK. In /proc/net/nf_conntrack you see the connection marked as state
ESTABLISHED, too.

Furthermore we tried to enter a static arp entry for the virtual cluster
ip on the client node, that points to the multicast mac. Same result.
But if you statically enter the real mac of the cluster node's interface
on the client node, see there, it works perfectly.

Is that possible a bug in conntrack?

Best,
Martin





-------------- next part --------------
A non-text attachment was scrubbed...
Name: client.pcap
Type: application/octet-stream
Size: 3775 bytes
Desc: not available
Url : /pipermail/netfilter-devel/attachments/20070719/65277d82/client.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: clusternode.pcap
Type: application/octet-stream
Size: 1354 bytes
Desc: not available
Url : /pipermail/netfilter-devel/attachments/20070719/65277d82/clusternode.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: firewall-script
Type: application/x-shellscript
Size: 3850 bytes
Desc: not available
Url : /pipermail/netfilter-devel/attachments/20070719/65277d82/firewall-script.bin
-------------- next part --------------
Jul 18 20:34:38 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36859 DF PROTO=TCP SPT=2002 DPT=80 SEQ=3177144544 ACK=933916978 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:34:48 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36866 DF PROTO=TCP SPT=2002 DPT=80 SEQ=3177144544 ACK=933916978 WINDOW=17520 RES=0x00 ACK FIN URGP=0 
Jul 18 20:34:51 cltestn2 kernel: Firewall DROP IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=36867 PROTO=UDP SPT=137 DPT=137 LEN=58 
Jul 18 20:34:52 cltestn2 kernel: device eth0 left promiscuous mode
Jul 18 20:34:52 cltestn2 kernel: device eth0 left promiscuous mode
Jul 18 20:34:52 cltestn2 kernel: Firewall DROP IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=36871 PROTO=UDP SPT=137 DPT=137 LEN=58 
Jul 18 20:34:53 cltestn2 kernel: Firewall DROP IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=36872 PROTO=UDP SPT=137 DPT=137 LEN=58 
Jul 18 20:35:01 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36908 DF PROTO=TCP SPT=2002 DPT=80 SEQ=3177144013 ACK=933916978 WINDOW=17520 RES=0x00 ACK PSH FIN URGP=0 
Jul 18 20:35:02 cltestn2 kernel: Firewall ESTABLISHED IN= OUT=eth0 SRC=172.16.29.200 DST=172.16.29.211 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=2002 SEQ=933916977 ACK=3177144013 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jul 18 20:35:02 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36909 DF PROTO=TCP SPT=2002 DPT=80 SEQ=3177144545 ACK=933916978 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:35:07 cltestn2 kernel: device eth0 entered promiscuous mode
Jul 18 20:35:07 cltestn2 kernel: device eth0 entered promiscuous mode
Jul 18 20:35:09 cltestn2 kernel: hash=1 ct_hash=1 responsible
Jul 18 20:35:09 cltestn2 kernel: hash=1 ct_hash=1 responsible
Jul 18 20:35:09 cltestn2 kernel: Firewall ACCEPT IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=36964 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997445 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0 
Jul 18 20:35:09 cltestn2 kernel: Firewall ESTABLISHED IN= OUT=eth0 SRC=172.16.29.200 DST=172.16.29.211 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=2003 SEQ=990544413 ACK=905997446 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jul 18 20:35:09 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36965 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997446 ACK=990544414 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:35:09 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36966 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997446 ACK=990544414 WINDOW=17520 RES=0x00 ACK PSH URGP=0 
Jul 18 20:35:12 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36968 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997446 ACK=990544414 WINDOW=17520 RES=0x00 ACK PSH URGP=0 
Jul 18 20:35:13 cltestn2 kernel: Firewall ESTABLISHED IN= OUT=eth0 SRC=172.16.29.200 DST=172.16.29.211 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=2003 SEQ=990544413 ACK=905997446 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jul 18 20:35:13 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36969 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997977 ACK=990544414 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:35:18 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36972 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997446 ACK=990544414 WINDOW=17520 RES=0x00 ACK PSH URGP=0 
Jul 18 20:35:19 cltestn2 kernel: Firewall ESTABLISHED IN= OUT=eth0 SRC=172.16.29.200 DST=172.16.29.211 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=2003 SEQ=990544413 ACK=905997446 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jul 18 20:35:19 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36973 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997977 ACK=990544414 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:35:30 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36976 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997446 ACK=990544414 WINDOW=17520 RES=0x00 ACK PSH URGP=0 
Jul 18 20:35:31 cltestn2 kernel: Firewall ESTABLISHED IN= OUT=eth0 SRC=172.16.29.200 DST=172.16.29.211 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=2003 SEQ=990544413 ACK=905997446 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jul 18 20:35:31 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36977 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997977 ACK=990544414 WINDOW=17520 RES=0x00 ACK URGP=0 
Jul 18 20:35:31 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=571 TOS=0x00 PREC=0x00 TTL=128 ID=36978 DF PROTO=TCP SPT=2002 DPT=80 SEQ=3177144013 ACK=933916978 WINDOW=17520 RES=0x00 ACK PSH FIN URGP=0 
Jul 18 20:35:32 cltestn2 kernel: Firewall ESTABLISHED IN=eth0 OUT= MAC=01:00:5e:00:00:20:00:04:76:13:60:1c:08:00 SRC=172.16.29.211 DST=172.16.29.200 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=36983 DF PROTO=TCP SPT=2003 DPT=80 SEQ=905997977 ACK=990544414 WINDOW=17520 RES=0x00 ACK FIN URGP=0 
Jul 18 20:35:35 cltestn2 kernel: device eth0 left promiscuous mode
Jul 18 20:35:35 cltestn2 kernel: device eth0 left promiscuous mode


More information about the netfilter-devel mailing list