no netfilter debugging?
Alistair at nerdnet.ca
Thu Apr 28 02:38:31 CEST 2005
On April 27, 2005 02:37 pm, Daniel Wittenberg wrote:
> I'm still trying to find out why NAT stops working after awhile, so I'm
> again going to see if anyone has a way to debug this. I'm not asking
> for people to debug it for me, I'll do the leg-work, but how do you
> debug at the kernel-level what netfilter is doing with packets? The
> firewall/load balancing works for awhile then suddenly stops working,
> could be 5 minutes, could be 5 hours...I am at a loss as to what causes
> it and nothing in dmesg/syslog for errors. Just flushing the rules and
> routing tables and re-configing fixes the problem so I'm guessing there
> is some limit it is hitting, but with nothing being logged I don't know
> where to start. I'm not sure if some of the kernel debugging tools can
> be limited to only view netfilter activity, but I think that's what I'll
> need to figure this out.
y'know -- I've seen this thread a bit and am wondering ...
1) Just an aside to think about:
You mention load balancing here... I can't recall off the top of my head if
its inbound LB or inbound LB, but I have to ask, do you have a pppoe link in
there? I have to ask, I've a cable modem and a DSL modem. The DSL is my
'service' port since the upstream doesn't filter ports, whereas the cable
link does, so my DNS and mail servers use the DSL side only. Periodically
the DSL stream provider has fits, and does weird things to their routing
which will result in my service side connections hanging. This makes things
look like they've stopped dead for a while until the DNS comes back. You
would be astonished at the number of processess that need 'gethostbyname' ..
even in the windows world.
You want to debug? -- LOG is your first step in debugging. Put a log rule at
the top and the bottom of each chain. --log-prefix is your friend here. Add
a cron or at process that dumps the state of /proc/net/conntrack and slabinfo
(you might want to filter that output to be relevant) and perhaps something
like netstat -an|grep WAIT (my favourite tool when debugging firewalls that
are pooching out -- if yer being too aggressive on filtering you could well
end up with sockets in all sorts of bad WAIT states.) all into
(syslog/messages/iptables/debug) logs.... make sure you get timestamps on
these. Make SURE you LOG everything your dropping as well... i.e.
insert a LOG rule before each drop statement
You will have HUGE logfiles after a very short period of time, however once
you get to the point where the firewall is stopping you will have a timestamp
and a chance to munge through the cruft and find the anomaly
Iptables components *can* have debug statements. Some need only to have the
DEBUG set in the source file, some need to have the DEBUG macros built. -- I
leave this to the Developers myself, but I know that several of the optional
modules to iptables have debug macros already. Read the source luke. Turn
em on and rebuild the kernel and modules. -- and be prepared to be spammed.
I could go on, but the above should provide you with a solid starting ground.
Once you *have* a hang, all those logs could come together to point to the
problem. There are those who can quickly review the output of the above and
give you a solid answer. You should sanitize the logged data by *ONLY*
removing data that could indicate your specific host/network and
usernames/etc ... in this case I STRONGLY reccomend that the post that you
send at that point have the following data:
1) kernel version/iptables version, compiler version
2) a visual layout of the network (inbound connections firewall location,
outbound connections and where what bits are i.e. firewall here, server a
there etc etc...)
3) the results of iptables -L -n -v -x --line-numbers
iptables -L -n -v -x --line-numbers -t nat
iptables -L -n -v -x --line-numbers -t mangle
ip route (or route -n)
and the logfiles as indicated above with a good period of time prior to and
after the halt.
Bonus points: *cough* gzip *cough* and a host where you could post the logs
would be a *really* good idea since you could then remove what might be
sensitive data from the net.
(just my 3 cents worth after a long day explaining what iptables is to a
More information about the netfilter