I now have a profound appreciation for BSD.
Yesterday, our pfSense firewall at 1102 Grand suddenly went silent and the panel on WhatsUp for that site went all red. Not good. I went down to the cage after small group last night and found the machine to have just simply locked up cold. I suspect hardware, since pfSense/BSD didn’t log a thing about it going dark.
It became quite clear that this setup was… suboptimal. Clif‘s shooting for 99.99% on this new setup. I can’t be racing off to the datacenter every time the firewall machine decides to take a holiday from reality. Brian and I quickly determined that we needed not only a remote power control unit, but some sort of high-availability solution that wasn’t going to empty our wallet like a pair of NSA 4500s would. (sorry Mark, we simply don’t have that kind of money) We already had a spare, identical machine at the datacenter doing duty as a hardware spare and development server, and another one just like it in inventory at the Central Campus. I grabbed the extra machine and went back 1102 Grand for the second time in 12 hours, with a quick stop at Micro Center for some cheap NICs and a red crossover cable.
Fortunately, pfSense has high availability capability built in, thanks to BSD’s CARP and pfSync. CARP allows me to set up virtual IPs on both firewalls and synchronize between them with pfSync. The extra NICs were for a dedicated sync/heartbeat link between the two boxes. I’m still a little fuzzy on the technical details of how this works, but it works… Convergence/Failover time is 3 seconds or less, and everything is synchronized between the two machines, including state and session information. I had it all set up, and hit the reset button on the primary firewall… and nary a ping was dropped.
The setup involves giving each machine its own LAN and WAN addresses (and a unique address on the other zones as well) and then creating a CARP virtual IP on that interface. The virtual IP is the one used as the gateway and as the NAT address. All rules and configs (including IPSEC Tunnels!) is mirrored on the second box.
Reference material:
Excellent documentation on the process from the folks at Countersiege.
Some background from the OpenBSD folks.
A few good tips here. These proved to be crucial.