Bandwidth woes

Given Kansas City’s central location on the IP backbones that traverse the US, it should be a heck of a lot easier than it is to get some semblance of decent bandwidth at the church (and other locations). Sadly, the best we can do is 1.5Mbit DSL (even if the line has been tested from our demarc to support 7.5Mbit, AT&T refuses to sell us any more than 1.5Mbit because their computers say that’s the best they can do) or bonded T1s.Even then, there’s a finite limit to what what we can achieve with that, because we have a finite amount of copper feeding the building.

Anything else involves construction costs in the $50K and up range. Even amortized over a 5-year contract, that still tacks on nearly $1000 to the monthly bill. Point-to-point wireless to somewhere with fiber would cost almost as much.

It amazes me that in one of the more affluent suburbs in the country, the best we can do is no better than what you get in and around most small towns in rural western Kansas.

Fixing network priority in Windows

Recently, we made some changes to the DNS infrastructure on our public wireless networks which has had the unintended consequence of breaking things when our laptop users are plugged into the LAN and have their wireless active. Brian and I have wrangled with this in the office, but we simply turned off the wireless as a workaround.

What’s happening is that when connected to both networks, the wireless has a higher priority by default, and so it resolves DNS via that interface. This is problematic when trying to access an internal resource, because our DNS is set to have a default resolution to our website for *.cor.org. To complicate matters further, Arena behaves differently when you’re on the guest network (sends to a forms-based auth portal instead of using IE integrated authentication).

After much digging, I found out how to change interface priority. Here’s the process in XP screenshots (the process is similar in Vista):

1. Open your network connection properties (XP: Via control panel or right-click on Network Places, then select Properties. Vista: Go to Network and Sharing Center and select “Manage Network Connections in the links on the left)

XP Network Properties

XP Network Properties

2. On the menu bar (press Alt to show it in Vista), Select Advanced, then “Advanced Settings”

Advanced Network Properties Dialog (XP)

Advanced Network Properties Dialog (XP)

3. Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.

4. Hit OK, and you’re good to go.

High-Availability firewall on the cheap!

I now have a profound appreciation for BSD.

Yesterday, our pfSense firewall at 1102 Grand suddenly went silent and the panel on WhatsUp for that site went all red. Not good. I went down to the cage after small group last night and found the machine to have just simply locked up cold. I suspect hardware, since pfSense/BSD didn’t log a thing about it going dark.

It became quite clear that this setup was… suboptimal. Clif‘s shooting for 99.99% on this new setup. I can’t be racing off to the datacenter every time the firewall machine decides to take a holiday from reality. Brian and I quickly determined that we needed not only a remote power control unit, but some sort of high-availability solution that wasn’t going to empty our wallet like a pair of NSA 4500s would. (sorry Mark, we simply don’t have that kind of money) We already had a spare, identical machine at the datacenter doing duty as a hardware spare and development server, and another one just like it in inventory at the Central Campus. I grabbed the extra machine and went back 1102 Grand for the second time in 12 hours, with a quick stop at Micro Center for some cheap NICs and a red crossover cable.

Fortunately, pfSense has high availability capability built in, thanks to BSD’s CARP and pfSync. CARP allows me to set up virtual IPs on both firewalls and synchronize between them with pfSync. The extra NICs were for a dedicated sync/heartbeat link between the two boxes. I’m still a little fuzzy on the technical details of how this works, but it works… Convergence/Failover time is 3 seconds or less, and everything is synchronized between the two machines, including state and session information. I had it all set up, and hit the reset button on the primary firewall… and nary a ping was dropped.
The setup involves giving each machine its own LAN and WAN addresses (and a unique address on the other zones as well) and then creating a CARP virtual IP on that interface. The virtual IP is the one used as the gateway and as the NAT address. All rules and configs (including IPSEC Tunnels!) is mirrored on the second box.

Reference material:

Excellent documentation on the process from the folks at Countersiege.

Some background from the OpenBSD folks.

A few good tips here. These proved to be crucial.

Turn the radio on!

(apologies to Randy Travis for lifting a title)

On Friday, our vendor came out to replace the radio on the Southcreek end of our wireless link. (More on that at Clif’s Blog). Long story short, we improved the income side of the link budget by about 16dB.

Got this done just in time for a big rainstorm on Saturday, followed by sloppy wet driving snow on Sunday (attendance was way down, partly due to the weather. Some churches even canceled service. Well, sort of.) Even Kansas City International Airport had its longest closure in history because they couldn’t keep the runways clear long enough. We Canadians are amused by this notion.

Since we had just gotten a shiny new radio and antenna on the Southcreek end, I was curious to see how the link was performing in the snow. I fired up WhatsUp and checked my wireless status page. Both bridges showed more or less the same thing:


(Time of day is along the X-axis, and the Y-axis is received signal level (RSL) in hundredths of a dB, so -3100 is -31dB – due to a firmware update, it only reports in whole dB now, probably because the fractional numbers weren’t nearly as accurate as they were precise )

The pattern struck me as intriguing, because precipitation generally looks a little different, as demonstrated by Saturday’s rainstorm (you can also see the beginning of the snow on the far right):

After checking a few weather sites, I discovered that the downward slope at 6:00 correlated to the beginning of the snow. I was beginning to suspect that at least one of the radomes was plastered in snow. We’d just gotten back from church, where the wind was blowing pretty hard from the northwest, and the Central Campus end was facing almost directly into the wind at the top of the building. I asked my wife if I could run back and do a little weekend science. After realizing that this sort of thing was part of what she signed up for when she married a geek, she sent me on my way with the camera (thank you honey, I love you! *smooch*)

I stopped by the Southcreek office first, and realized that the blue Bridgewave logo on the radomes was going to be very helpful at determining accumulation. This is what Southcreek looked like:

(apologies for the grainy picture, it was taken from about 100 feet away at max digital zoom and then cropped):

Unsurprisingly, there was no significant accumulation on the Southcreek radio, as the radome was facing downwind. This is what the weather looked like towards the other end of the link:

I drove over to the church, where the conditions looked like this:

Notice that the snow is plastered on one side of the trees. The CC radio is facing that direction.

I found a radio and got a hold of George (on the facilities team, also does desktop support for us one day a week) to let me onto the roof. George looked at me funny and wondered why I wanted up on the roof in this craptacular weather. After a brief explanation, he joined me (and wanted to see for himself, too – George is a geek at heart). I get up on the roof, and do a little skating (roofing membrane is nice and slick when wet, never mind when covered in a few inches of sloppy wet snow!)

Sure enough, here’s what the radome looked like:

It was pretty clear what was causing our 30dB signal loss (the link was still up, with about 10dB to go). George went off to find something to clean off the snow (it’s about 15′ from where we were standing, and we didn’t have a ladder). While George was off playing MacGyver, I got to thinking that the snow probably wasn’t stuck on very well, and that some sort of jarring impact might knock it off. If only I had something to throw at it… Like, say, a snowball. My concern was that the snowball would stick to the radome and REALLY attenuate the signal, but I figured this stuff was wet and slushy enough to form into a ball, but was too wet to actually stick to anything (it was above freezing the whole time). So I started chucking snowballs at a piece of gear that costs about the same as a decent new car (I love my job!). On the third try, I made solid contact just below the logo, and the sheet of snow came sliding right off (look below the right loop of the logo for the point of impact):


(by the time I actually got the picture taken, some more snow had accumulated on the radome. Did I mention it was snowing hard?)

I went down to a computer to check on the signal level. Sure enough, the link improved a bunch. (I’ll repost the image here so you don’t have to scroll all the way to the start of the post.) The snowball caused the sharp vertical spike on the right side of the graph. The picture was taken about the spot where it dropped back down a few DB:

I headed back for the roof and found George had MacGyvered a pole from an extendable dusting wand and a wooden broom handle, held together with packing tape. I climbed back up onto the roof and was able to reach the radome with George MacGyver’s snow brush. Cleaning it off gained me a few more dB (second, smaller vertical spike on the graph):

As you can see on the graph, some more snow started accumulating, and then the snow stopped and started melting off. By mid-afternoon, the sun had come out we were back up to our normal signal levels, and there was little evidence left around town that we’d even had a snowstorm. We went from this, where it’s snowing sideways…

…to a beautiful sunny day in a matter of hours. I’m glad I didn’t bother shoveling my driveway, as it had melted clear by the time my wife and I got back from the movies (we went to see Jumper. Good flick, but left a lot of unanswered questions — sequel, anyone? — as well as leaving me with lingering nausea from the jumpy camera work)

I haven’t heard what the attendance was like at the 5:00 service. Morning services were sparse due to weather, but Rev. Junius Dotson from Saint Mark UMC in Wichita was our guest preacher this week and preached a great sermon (Adam is off in Colorado enjoying the real snow with the high schoolers). I hope a bunch of folks got to experience Rev. Dotson at the evening service. The man just has style.

And now, for the ADD folks that lost me about 6 paragraphs ago, here’s a nice little summary:

Wi-Spying at Leadership Institute…

I’ve been keeping an eye on our new wireless (SonicPoints!), and noticed something interesting… This 60-minute image was captured at 10:17. Can you guess what time people started coming into the sanctuary?

I’m guessing that it’s probably a combination of bluetooth from all the cell phones in the room along with different RF dispersion patterns because the room filled up with a lot of very wet objects (humans) which absorbs 2.4GHz quite readily.