What happens during a slashdotting?

November 5, 2008 |   |  Comment  |  networking

Well, OK, it wasn’t Slashdot that was the culprit this time, but rather the pro blog Consumerist (if we’re a megachurch, does Consumerist count as a megablog? It claims nearly 3 million unique visitors a month)

Last week, Clif posted about his experience at Best Buy. Seems the folks at Gawker Media got wind of the story (Best Buy is a perennial favourite target of theirs) and posted it at 10:21am Eastern, 9:21 in KC. Here’s what happened to our WordPress server:

Apache Processes, November 5

Apache Processes, November 5

Web Datacenter traffic, November 5 (% of 10Mbps link)

Web Datacenter traffic, November 5 (% of 10Mbps link)

Wow. I noticed the odd traffic behaviour (that particular server gets very little traffic most of the time) when I got in the office, and called a few folks to see if they’d done anything that would cause this. When that came up empty, I started looking at the access logs on the server and noticed a lot of referrer traffic from Consumerist. I threw AWstats onto the server to grok the apache logs. At posting time, Clif’s blog post had seen around 7000 visitors. Apache peaks out at a point due to the MaxClients directive, in order to keep the CPU from saturating and killing the site.

It’s always fun to see new an interesting traffic patterns. It’s very helpful to have active monitoring to tell us when things leap outside the bounds of normalcy.

My 5-year-old is very perceptive…

October 20, 2008 |   |  Comment  |  Family

She just built a large structure out of Lego, and it has a bell.

She was explaining that it’s a school, called "Zoo" (it has a piece above the door that says so, presumably from a zoo set), and that it was a high school.

Sounds like she has a pretty good grasp on what high school really is.

Bandwidth woes

October 20, 2008 |   |  Comment  |  networking

Given Kansas City’s central location on the IP backbones that traverse the US, it should be a heck of a lot easier than it is to get some semblance of decent bandwidth at the church (and other locations). Sadly, the best we can do is 1.5Mbit DSL (even if the line has been tested from our demarc to support 7.5Mbit, AT&T refuses to sell us any more than 1.5Mbit because their computers say that’s the best they can do) or bonded T1s.Even then, there’s a finite limit to what what we can achieve with that, because we have a finite amount of copper feeding the building.

Anything else involves construction costs in the $50K and up range. Even amortized over a 5-year contract, that still tacks on nearly $1000 to the monthly bill. Point-to-point wireless to somewhere with fiber would cost almost as much.

It amazes me that in one of the more affluent suburbs in the country, the best we can do is no better than what you get in and around most small towns in rural western Kansas.

Fixing network priority in Windows

October 10, 2008 |   |  3 Comments  |  networking, Wireless

Recently, we made some changes to the DNS infrastructure on our public wireless networks which has had the unintended consequence of breaking things when our laptop users are plugged into the LAN and have their wireless active. Brian and I have wrangled with this in the office, but we simply turned off the wireless as a workaround.

What’s happening is that when connected to both networks, the wireless has a higher priority by default, and so it resolves DNS via that interface. This is problematic when trying to access an internal resource, because our DNS is set to have a default resolution to our website for *.cor.org. To complicate matters further, Arena behaves differently when you’re on the guest network (sends to a forms-based auth portal instead of using IE integrated authentication).

After much digging, I found out how to change interface priority. Here’s the process in XP screenshots (the process is similar in Vista):

1. Open your network connection properties (XP: Via control panel or right-click on Network Places, then select Properties. Vista: Go to Network and Sharing Center and select “Manage Network Connections in the links on the left)

XP Network Properties

XP Network Properties

2. On the menu bar (press Alt to show it in Vista), Select Advanced, then “Advanced Settings”

Advanced Network Properties Dialog (XP)

Advanced Network Properties Dialog (XP)

3. Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.

4. Hit OK, and you’re good to go.

Say Whaaaaaaaa?????

August 20, 2008 |   |  2 Comments  |  IT

The last few days have been rather stressful.. Our shiny new web infrastructure at COR has been throwing major temper tantrums, which means I’ve been rather busy of late. Today, it melted down half a dozen times after I thought I’d fixed it. Each time it did something different. (and if that wasn’t enough excitement, our upstream provider had a BGP issue this morning that knocked their entire customer base off the web for about 5 minutes)

All you folks that hit our website, thank you for your patience. These have been trying times.

This morning, I noticed something very odd. And again this evening as I’m migrating the data to a new server.

root@corweb1:/content/sites# uptime
11:35:56 up -24855 days, -3:-14,  4 users,  load average: 0.00, 0.00, 0.00

Negative uptime??? What?

root@corweb1:/content/sites# date
Wed Aug 31 11:50:17 CDT 1955

Perhaps running apt-get install flux-capacitor wasn’t such a good idea, as the machine seems to be performing on about the level of a computer from 1955.

It’s become pretty obvious that something is very ill on that box, and I think it’s time to ditch VMWare Server for ESXi. Until then, we’re moving the servers over to the bare metal on the other box (which includes the blog server, it’s already been moved).

Last login: Wed Aug 31 11:44:51 1955 from XXXX.kc.res.rr.com

No wonder I feel old.

Arena Check-In

August 1, 2008 |   |  1 Comment  |  Church IT

Now that a lot of the back end of our Arena Check-in system is in place and ready to go, I’ve been focusing my efforts on the fun part… Themes!

This school year’s theme is “Go Fish”, so I set about seeing if I could come up with something fun, without hiring an external designer/illustrator to do something for us. Here’s what I’ve got so far:

Main Screen

Phone Lookup Screen

Family Screen

Still a work in progress, haven’t done the buttons yet (as anyone familiar with the default Arena theme has already spotted), but it’s coming along.

I’ve got 4 different scenes to wotk with for the fish, and the original art is vector-based, so I can move fish about as needed to get out of the way of text – I think I can even make them swim from one illustration to the other via the miracle of the Windows clipboard.

The artwork is from iStockPhoto, from a designer who goes by the username totallyjamie. The font is Chaloops Medium from Chank!.

What I did on my summer vacation!

July 24, 2008 |   |  1 Comment  |  Family

This past weekend had the family taking a quick trip out to Connecticut to visit my grandmother for her 90th birthday party, a surprise celebration my dad and aunts have benn putting together for a couple of years now.

We flew up to Hartford on Saturday and rented a car to drive down to New Haven/Orange where I scored a room at the Courtyard Marriott for about 35% of the regular rate, thanks to Moreno’s tips on how to make Priceline work for you.

We met my dad who was holding a spot in line at Frank Pepe’s while the rest of the gang showed up. After some waiting, my aunt and grandmother showed up. I walked across the street and asked my grandmother if I could help a little old lady across the street 🙂 After about half a second, it registered who asked her, and I got a big hug for my troubles after we got to the sidewalk. She had no idea we were coming to visit. A little later on, my brother showed up as well, on his way to Dublin. We got a table and had some great apizza. After the meal, our waitress got up and announced to the whole place that it was Bev’s 90th birthday and had them all sing. My grandmother told them all that her brother was one of the early customers of the place, and that she and her cousin were sitting in nearly the exact same spot when the radio came on to announce that President Roosevelt had died. Turns out the waitress was Frank Pepe’s granddaughter and that it’s still very much a family business, and they were honored to still have some one of their early customers. My grandmother promised to be back for her 100th.

The following day, more extended family gathered for another surprise at the Norm Bloom & Sons quay. My grandmother had been told that we were going to visit a “maritime museum”. After a little resistance from her and some convincing on my aunt’s part that she REALLY needed to come with us, she saw the extended family all wearing matching T-shirts for the occasion. As she’s digesting this and more family are trickling in, we take her around the back of the building where she sees the Catherine M. Wedmore, her grandfather’s oyster boat, set up for a Sunday afternoon cruise and picnic. The last time she’d seen the boat was 1951. It was sold by the Wedmores to the Blooms in the 1960s. 84 years after her maiden voyage, she’s still working the bay dredging for clams and oysters.

We have family footage of a similar sunday afternoon picnic, from 1940.

A reporter from the Stamford Advocate came along and wrote up a story.

Once out in the bay, we dropped one of the dredges and scooped up a bunch of fresh oysters, which were shucked and eaten. In one shell we found a small crab about the size of a dime rooming with one of the oysters. The girls had lots of fun meeting their 2nd cousin Adam (my Cousin Jeff’s son), and my cousin Catherine and her hubby gave my grandmother one more surprise, that she was going to have a fourth great-grandchild in February.

Lots of pictures were taken. Pics to follow.

I knew this day would come….

July 7, 2008 |   |  2 Comments  |  Family

My eldest, Faith (who will be *gasp* FIVE next month) showed from a very early age (6 months?) that she was probably going to be one of these extreme-sports adrenaline junkies.

Today, she proved it. She begged us to go on the EuroBungy ride at the Stampede. 10 bucks later, she was harnessed up:

And not too long after, she was flying:

She absolutely loved it, and followed it up with a trip on one of the kiddie coasters. She’ll be moving up to the big stuff before too long. We’ve been saying for years that she’ll be a coaster junkie by the time she’s six.

CITRT expands northward (and westward)!

July 7, 2008 |   |  Comment  |  Church IT

I had a great chat this morning with Gregg Hatton-Fearnley, the IS/IT director at Centre Street Church here in Calgary.

The main purpose of my meeting with Gregg was to introduce myself and tell him about CITRT. CSC is the second largest church in Canada (the largest is either in Toronto or Winnipeg, he couldn’t recall which). Like most people who hear about CITRT, he was quite enthusiastic about the idea, and was bemoaning the lack of funds in his budget to go to the fall event at Seacoast. I have high hopes of seeing him at spring RT, though.

CSC has got a great facility which, as most of us have experienced, they practically had outgrown the day they opened the building. I was very impressed with Gregg’s openness and transparency about some of the big challenges he faces – he shared a lot for a first date! 🙂

Unsurprisingly, his biggest hurdles aren’t so much technological as they are human. (Has anyone else had problems getting the Worship Arts people to play nice on the network? Yeah, thought so. Wasn’t that the whole point of MinistryTECH? )

The other really big challenge CSC faces is personnel. The Calgary job market is tremendously competitive, with entry-level fast food workers fetching a starting wage of $15/hour, and a low-level network admin commands a 6-figure salary. It’s hard enough finding people to work for church wages in a sane job market, it’s got to be ridiculously hard to do in Calgary, especially when Big Oil is snapping up all the IT people it can, and for huge money.

CSC is also a recent addition to Fellowship One’s growing list of customers, and I got to experience some quirks of the check-in system on Saturday night when Andrea and I went to check out.

CITRT is truly an international bunch now… We’ve got Canadians and Australians on board too. I hope to see Gregg on IRC or the Talkcast soon 🙂

Naturally, no church visiti is complete without pictures from the facility:

This is the projector set for one of the side screens. This thing is a monster. Hard to tell from the picture, but each one is about the size of a coffin. These are mounted in the tech booth at the back of the 2400-seat sanctuary. The throw on these is a good 200 feet, possibly more.

CSC's projectors for one of the side screens

The video control room:


Video Switcher:

A decorative waterfall in the narthex. The sculpture in the middle can be removed to turn it into a baptismal pool. The “chandeliers” nearby each have about 32 dimmer channels and are built to support stage lighting as well.

The other side of the narthex: