*Very* strange netfilter or conntrack bug??
Adam Ierymenko
api@clearlight.com
Sun, 17 Jun 2001 16:12:29 -0400
(I posted this already to netfilter-devel but didn't get any
good responses, so I'm posting it to the main list.)
-----
I think this is a netfilter bug, so I am posting this here. If I am
posting this in the wrong place or if you don't think this is a
netfilter bug feel free to direct me elsewhere.
To be more accurate, I am suspicious that this is an IP
conntrack bug.
This is without a doubt the strangest networking behavior I
have ever seen in my life. I have been using Linux since about
1993 and my previous job was with an ISP, so I have seen a
lot and I generally know what I am doing.
We have a configuration here at the company I work for in
which we have two Linux machines with a T1 line between
them. One of them is at a colocation facility and the other is
at our office. Both machines have LMC PCI T1 cards in them.
The machine at the colocation is a PII-400 with 128mb of
RAM running Linux 2.4.5 at the moment. The router at this
end is a little old pentium box with 2.2.18 on it. I believe the
bug or problem lies at the 2.4.5 end since it seems to have
started since we upgraded from 2.4.3 to 2.4.4 (and persisted
when we upgraded to 2.4.5). I don't want to downgrade to
2.4.3 since 2.4.3 has a potential security problem.
Basically the 2.4.5 router box whose name is 'gypsy' has
iptables, ip conntrack, and 'the works' compiled into it. It
needs all this since it both routes our office IP block to the
net and routes between a private 'fake IP' block at the
colocation and our office for secure access. The routing
between the 'fake ip' block and the office is done via
IP masquerading so it needs connection tracking.
Since we upgraded to 2.4.4 the T1 line has seemed "choppy"
even when there is virtually no traffic on it. I suspected this
was a problem with the line so I had Cincinnati Bell do a
loopback test and some other rigorous line tests and they
found no problem. I then did some tests myself and came
up with... well...
cue x-files music...
---------
api@abyss:~$ ping outer
PING outer.xactcommerce.com (209.50.98.82): 56 data bytes
64 bytes from 209.50.98.82: icmp_seq=0 ttl=254 time=5.3 ms
64 bytes from 209.50.98.82: icmp_seq=1 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=2 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=3 ttl=254 time=3.8 ms
--- outer.xactcommerce.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 3.8/4.1/5.3 ms
api@abyss:~$ ping outer
PING outer.xactcommerce.com (209.50.98.82): 56 data bytes
64 bytes from 209.50.98.82: icmp_seq=0 ttl=254 time=305.2 ms
--- outer.xactcommerce.com ping statistics ---
6 packets transmitted, 1 packets received, 83% packet loss
round-trip min/avg/max = 305.2/305.2/305.2 ms
api@abyss:~$ ping outer
PING outer.xactcommerce.com (209.50.98.82): 56 data bytes
64 bytes from 209.50.98.82: icmp_seq=0 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=1 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=2 ttl=254 time=3.8 ms
--- outer.xactcommerce.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.8/3.8/3.8 ms
api@abyss:~$ ping outer
PING outer.xactcommerce.com (209.50.98.82): 56 data bytes
64 bytes from 209.50.98.82: icmp_seq=0 ttl=254 time=283.6 ms
64 bytes from 209.50.98.82: icmp_seq=1 ttl=254 time=305.9 ms
64 bytes from 209.50.98.82: icmp_seq=2 ttl=254 time=291.0 ms
64 bytes from 209.50.98.82: icmp_seq=3 ttl=254 time=290.6 ms
--- outer.xactcommerce.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 283.6/292.7/305.9 ms
--------
All I am doing is restarting ping. There is no traffic to speak of
on our T1 right now. I am doing this from my desktop machine at
my office which is not even a router. I get the same sort of strange
behavior if I do this from either one of the routers over the T1.
Yeah, you are seeing it right... simply restarting ping generates
wildly different latencies and those latencies will persists until ping
is restarted... at which time some other latency will occur.. sometimes
low or sometimes high.
Not only that.. it gets wierder...
---------
Window 1:
64 bytes from 209.50.98.82: icmp_seq=8 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=9 ttl=254 time=3.9 ms
64 bytes from 209.50.98.82: icmp_seq=10 ttl=254 time=3.8 ms
64 bytes from 209.50.98.82: icmp_seq=11 ttl=254 time=3.7 ms
64 bytes from 209.50.98.82: icmp_seq=12 ttl=254 time=3.7 ms
Window 2:
64 bytes from 209.50.98.82: icmp_seq=0 ttl=254 time=230.1 ms
64 bytes from 209.50.98.82: icmp_seq=1 ttl=254 time=244.3 ms
64 bytes from 209.50.98.82: icmp_seq=2 ttl=254 time=237.2 ms
64 bytes from 209.50.98.82: icmp_seq=3 ttl=254 time=253.2 ms
64 bytes from 209.50.98.82: icmp_seq=4 ttl=254 time=235.8 ms
---------
This came from two different ping processes running concurrently
on the same box pinging 'outer' over the T1.
I get the same results from a Windows 2000 box in our office too.
I was also able to replicate the different results from two different
ping processes running concurrently on the Win2000 box.
Obviously one of the routers is doings something incomprehensibly
wierd. Like I said, I think this is the 2.4.5 machine since the 2.2.18
machine has not changed in a very long time and this behavior
seemed to start with the 2.4.3 -> 2.4.4 upgrade.
On other types of connections such as TCP connections this seems
to produce 'choppiness' and wildly differing download times for
different connections to the same site. SSH and telnet sessions are
'choppy' with wildly differing latency.
The reason why I think this is a netfilter/conntrack bug and not a
T1 driver bug or a net core bug (I could be wrong) is because the
only way I can conceive of this behavior occurring is that
something on the router has got to be 'tracking' the fact that these
are different ping instances and producing the behavior for one
ping instance but not the other.
The same behavior also occurs when pinging the masqueraded
private IPs as the colocation facility or when pinging another
box's outside IP at the colocation.
Here is our configuration in depth:
gypsy - 2.4.5 - single PII-400 128mb RAM
- Gypsy has two outer IPs: 209.50.98.82 and 209.50.98.84.
- 209.50.98.82 is an alias
- 209.50.98.82 is the 'routing' ip while 209.50.98.84 is this
machine's identity as our mailserver.
- The T1 line is running via LMC T1 cards using the
1.34.15 WAN driver and Cisco HDLC framing.
- T1 is a standard point-to-point T1 line not a frame-relay.
- T1 has been exhastively checked for trouble
- It does masquerading between 216.23.24.* block over T1
and 10.1.1.* which is a 'private' lan at our colocation
facility that the web servers use to communicate with the
database server. We can route to 10.1.1.* from our end
and it will masquerade through to the private lan at the
colocation.
- It also does routing between the net and 216.23.24.*
- It runs iptables and ip_conntrack
inner - 2.2.18 - old pentium (120 I think) with 24mb RAM
- Inner has one IP - 216.23.24.65
- It's configuration is simple-- simple routing over T1
- Hasn't changed in a long time
- Same version of T1 driver but compiled against 2.2 kernel
Like I said, if this is the wrong place or you really don't think this
is a netfilter or conntrack bug, let me know. Also feel free to
forward this e-mail anywhere you want if you think you know
someone who should read this.
If there are any diagnostics you want me to do, let me know. I
would even be willing to create an account for one of the core
developers on the 'inner' router or 'gypsy' so they can see for
themselves. I suspect that someone might just want to see this
behavior for themselves since this is so incredibly strange. I
can't even concieve of how this is happening with ICMP PING
myself but I don't know IP in-depth enough to know for sure.