Linux, iptables, SNAT and Too Many Adapters
I've had some weird glitches with my primary firewall this week. After a reboot, certain SNAT rules just didn't... happen. Or, other SNAT rules did. And it always seemed to only impact SIP traffic.
For instance, after rebooting, SIP traffic from one internal phone server that's destined for another internal phone server at the other end of a VPN connection would spontaneously decide to SNAT itself to an external IP address, yet still go across the tunnel.
Or, SIP traffic destined for the outside would decide to route correctly out the right adapter, but leave the private address on it, ignoring the SNAT rule I could plainly see in iptables.
The fun part is that this was random, but always occurred directly after a reboot. Completely flushing and reloading the iptables ruleset, including the NAT and MANGLE tables did absolutely nothing for me. The only fix was another reboot and cross your fingers the problem didn't move to another target.
Let me go on to explain that it's a very complex server with 16 different IP addresses assigned across 4 network ports, and two different VPN tunnels. The final fix was literally to disable one physical network port not being currently used, and disable one virtual IP address that was no longer in use. Now everything is working perfectly.