Say Whaaaaaaaa?????

The last few days have been rather stressful.. Our shiny new web infrastructure at COR has been throwing major temper tantrums, which means I’ve been rather busy of late. Today, it melted down half a dozen times after I thought I’d fixed it. Each time it did something different. (and if that wasn’t enough excitement, our upstream provider had a BGP issue this morning that knocked their entire customer base off the web for about 5 minutes)

All you folks that hit our website, thank you for your patience. These have been trying times.

This morning, I noticed something very odd. And again this evening as I’m migrating the data to a new server.

root@corweb1:/content/sites# uptime
11:35:56 up -24855 days, -3:-14,  4 users,  load average: 0.00, 0.00, 0.00

Negative uptime??? What?

root@corweb1:/content/sites# date
Wed Aug 31 11:50:17 CDT 1955

Perhaps running apt-get install flux-capacitor wasn’t such a good idea, as the machine seems to be performing on about the level of a computer from 1955.

It’s become pretty obvious that something is very ill on that box, and I think it’s time to ditch VMWare Server for ESXi. Until then, we’re moving the servers over to the bare metal on the other box (which includes the blog server, it’s already been moved).

Last login: Wed Aug 31 11:44:51 1955 from XXXX.kc.res.rr.com

No wonder I feel old.

Turn the radio on!

(apologies to Randy Travis for lifting a title)

On Friday, our vendor came out to replace the radio on the Southcreek end of our wireless link. (More on that at Clif’s Blog). Long story short, we improved the income side of the link budget by about 16dB.

Got this done just in time for a big rainstorm on Saturday, followed by sloppy wet driving snow on Sunday (attendance was way down, partly due to the weather. Some churches even canceled service. Well, sort of.) Even Kansas City International Airport had its longest closure in history because they couldn’t keep the runways clear long enough. We Canadians are amused by this notion.

Since we had just gotten a shiny new radio and antenna on the Southcreek end, I was curious to see how the link was performing in the snow. I fired up WhatsUp and checked my wireless status page. Both bridges showed more or less the same thing:


(Time of day is along the X-axis, and the Y-axis is received signal level (RSL) in hundredths of a dB, so -3100 is -31dB – due to a firmware update, it only reports in whole dB now, probably because the fractional numbers weren’t nearly as accurate as they were precise )

The pattern struck me as intriguing, because precipitation generally looks a little different, as demonstrated by Saturday’s rainstorm (you can also see the beginning of the snow on the far right):

After checking a few weather sites, I discovered that the downward slope at 6:00 correlated to the beginning of the snow. I was beginning to suspect that at least one of the radomes was plastered in snow. We’d just gotten back from church, where the wind was blowing pretty hard from the northwest, and the Central Campus end was facing almost directly into the wind at the top of the building. I asked my wife if I could run back and do a little weekend science. After realizing that this sort of thing was part of what she signed up for when she married a geek, she sent me on my way with the camera (thank you honey, I love you! *smooch*)

I stopped by the Southcreek office first, and realized that the blue Bridgewave logo on the radomes was going to be very helpful at determining accumulation. This is what Southcreek looked like:

(apologies for the grainy picture, it was taken from about 100 feet away at max digital zoom and then cropped):

Unsurprisingly, there was no significant accumulation on the Southcreek radio, as the radome was facing downwind. This is what the weather looked like towards the other end of the link:

I drove over to the church, where the conditions looked like this:

Notice that the snow is plastered on one side of the trees. The CC radio is facing that direction.

I found a radio and got a hold of George (on the facilities team, also does desktop support for us one day a week) to let me onto the roof. George looked at me funny and wondered why I wanted up on the roof in this craptacular weather. After a brief explanation, he joined me (and wanted to see for himself, too – George is a geek at heart). I get up on the roof, and do a little skating (roofing membrane is nice and slick when wet, never mind when covered in a few inches of sloppy wet snow!)

Sure enough, here’s what the radome looked like:

It was pretty clear what was causing our 30dB signal loss (the link was still up, with about 10dB to go). George went off to find something to clean off the snow (it’s about 15′ from where we were standing, and we didn’t have a ladder). While George was off playing MacGyver, I got to thinking that the snow probably wasn’t stuck on very well, and that some sort of jarring impact might knock it off. If only I had something to throw at it… Like, say, a snowball. My concern was that the snowball would stick to the radome and REALLY attenuate the signal, but I figured this stuff was wet and slushy enough to form into a ball, but was too wet to actually stick to anything (it was above freezing the whole time). So I started chucking snowballs at a piece of gear that costs about the same as a decent new car (I love my job!). On the third try, I made solid contact just below the logo, and the sheet of snow came sliding right off (look below the right loop of the logo for the point of impact):


(by the time I actually got the picture taken, some more snow had accumulated on the radome. Did I mention it was snowing hard?)

I went down to a computer to check on the signal level. Sure enough, the link improved a bunch. (I’ll repost the image here so you don’t have to scroll all the way to the start of the post.) The snowball caused the sharp vertical spike on the right side of the graph. The picture was taken about the spot where it dropped back down a few DB:

I headed back for the roof and found George had MacGyvered a pole from an extendable dusting wand and a wooden broom handle, held together with packing tape. I climbed back up onto the roof and was able to reach the radome with George MacGyver’s snow brush. Cleaning it off gained me a few more dB (second, smaller vertical spike on the graph):

As you can see on the graph, some more snow started accumulating, and then the snow stopped and started melting off. By mid-afternoon, the sun had come out we were back up to our normal signal levels, and there was little evidence left around town that we’d even had a snowstorm. We went from this, where it’s snowing sideways…

…to a beautiful sunny day in a matter of hours. I’m glad I didn’t bother shoveling my driveway, as it had melted clear by the time my wife and I got back from the movies (we went to see Jumper. Good flick, but left a lot of unanswered questions — sequel, anyone? — as well as leaving me with lingering nausea from the jumpy camera work)

I haven’t heard what the attendance was like at the 5:00 service. Morning services were sparse due to weather, but Rev. Junius Dotson from Saint Mark UMC in Wichita was our guest preacher this week and preached a great sermon (Adam is off in Colorado enjoying the real snow with the high schoolers). I hope a bunch of folks got to experience Rev. Dotson at the evening service. The man just has style.

And now, for the ADD folks that lost me about 6 paragraphs ago, here’s a nice little summary:

Following the spam…

Mark Wade at CA recently posted a neat article about what happens when you respond to a spam message. Check out some of Mark’s other posts as well. Lots of good stuff there for our users, but in a forum where your typical end-user is not likely to be found.

I will occasionally chase down phishing messages (and populate with bogus data) just to see how elaborate the phish is, so I can warn our users. There have been a few recently that have been particularly well crafted, and thus pose a higher threat, as the deception is far more effective.

Just a reminder for your users, unless your bank is run by a bunch of idiots, they won’t e-mail you out of the blue. If they do, it’s time to change banks.