Here’s a neat new idea from Terry Storch and the folks at LifeChurch… It’s called “YouVersion” and launches this week. The Bible has just gone Web 2.0! I can see some really cool potential if I can use this from my smartphone and add notes and commentary during sermons, and if preachers can feed XML with tags and scripture references. Hello, mashups!
India-based tech support…
I’m really starting to loathe support calls to vendors.
I’m still scratching my head wondering who came up with the idea in the first place that it was a smart customer move to send technical support overseas, and have your customers try to explain complex technical issues over a bad phone connection to someone whose native language isn’t English, and who speaks English with a thick accent. I know how frustrating it is for non-anglophones to try and explain complex technical things to me, why did the industry suddenly decide it was a good idea to make this standard customer policy?
I can tell that a lot of these support agents (especially at Microsoft) are very technically skilled, but there’s the problem of accurately communicating the actual problem to them, which prolongs the process significantly (in the case of Microsoft, up to several hours!).
Support from SonicWall has taken a definite turn for the worse of late. After you get through the ticket dispatcher (who is usually based in the US) and have your ticket submitted (BTW, Sonicwall, how hard can it possibly be to get my e-mail address right? I spell it out for you every time and yet you still manage to make it undeliverable!), you end up with some second-line tech who still seems to be operating off a script. Any time you ask something remotely complex, it’s back into holdland while he goes and asks someone who actually knows what he’s doing (or she, especially at Microsoft, where the hardcore brains all seem to be women!)
As for VMWare, it’s always an adventure in global roulette. It’s a nice distraction from India, and their overseas support folks are usually extremely fluent in English.
I understand the need to offshore your support if you offer it for free, but when I (or my employer) pay big bucks for support contracts or on a per-incident basis, I expect something other than the lowest overseas bidder.
Microspeedlinking…
- The New York Times talks about Vanu Bose and his quest to make software-defined radios ubiquitous
- DHS is starting to look a lot like James Bond’s world…
On the road again…
I’m spending Labor Day weekend at Andrea’s dad’s place in Valparaiso, IN. Much needed grandkid time for him, mini-vacation for us.
We’re such church nerds. On Friday afternoon, we went up to Willow Creek Church for an informal tour of their facility with Mark Stanger, one of their techie guys. Andrea is a theatre nerd (she even has a degree in theatre!) and was duly impressed by what they had at WC. We also got a brief tour afterwards of the IT facilities from Brett (I missed his last name), one of my counterparts there. Technically cool, and professionally informative. It’s nice to see that they face many of the same challenges we do.
Today, we got a chance to go worship at Granger Community Church and break bread (and noodles!) afterwards with Jason Powell and his family. A good time was had by all, and the worship experience was phenomenal. They’re doing some really awesome stuff at GCC that we could definitely learn from. (I found it amusing that today’s message was about work/life balance, right after having had my vacation interrupted last night by our VMWare datastore filling up and causing the database server VM to go on strike due to lack of swap space)
One of the coolest things they have at GCC is the check-in/drop-off process for the kids. If you’re a regular, you check in, get your kids stickered, and then take them to the “launch area”, which is a big tube slide where the kids slide down and pop right into their classroom (unfortunately, Clara wasn’t quite old enough for those classrooms, so we had to cart her down the stairs (where there is a really cool fish tank). Faith went down her slide and popped out the mouth of a giant whale. They have cameras at the top and bottom so you can see them popping out the end and into the classroom. Faith was duly impressed. My immediate thought was that it’s a great way to deal with separation anxiety problems – the kids get a rush of fun, and immediately forget that they’ve just been taken away from mom and dad… and there’s no turning back. I think there’s a metaphor in there for the Christian life too 🙂
The worship experience was, in a word, WOW. They’ve done an incredible job at GCC to make the entire experience immersive and seamless. There’s smooth and logical integration of music, drama, and multimedia that all blend into the message. There’s some really cool technical wizardry going on behind the scenes, but you really don’t have a chance to focus on that, because the experience is so engaging.
On the way back, the kids snoozed, and we passed two things of note…
Valpo has a ski resort. OK, not really a resort, more of a lump in the terrain with a chairlift. To wit:
Where I come from, that would barely be worth bothering with as a sledding hill. Note the snowboarding halfpipe in the second picture.
The other thing that we saw was another Living Water(s) church, that is even smaller than the one Clif‘s wife planted:
And, while I’m posting nerdy pics from my phone, I finally got our rack in something close to its final configuration:
From top to bottom:
- Sony LIB-81A Tape library (8 slots, 1 AIT-3 drive)
- Galilee (DR Server) – Dell PE2650
- Buffalo TeraStation and TeraStation Pro (archival storage)
- Jericho (ESX Node) – Dell PE2950, 2xIntel 5160, 12GB
- Jerusalem (ESX Node) – Dell PE2950 2xIntel 5150, 12GB
- 8-port KVM Console
- Dell/EMC AX150i iSCSI SAN (1.5TB)
- UPS for the SAN
- Dell PowerVault 220 (12x73GB SCSI) (attached to Galilee for DDT backup staging)
- APC Smart-UPS 2200 (x2)
- Whitebox security controller PC (in the bottom of the rack)
This makes up almost our entire infrastructure except for telecom and networking.
Willow Creek Leadership Summit
Today is day 2 of the Willow Creek Leadership Summit. Naturally, I have pictures!
Off I go, into the wild blue yonder…
I’m currently sitting in the departure area at Kansas City International’s Terminal C awaiting my flight to Denver. This all came up rather suddenly last Friday afternoon when I got an E-mail from Bill English at MindSharp, offering me a seat at one of their SharePoint training events. The only one they had scheduled for Kansas City this year was the course for SharePoint Designer 2007, which Bill admitted probably wasn’t quite what I needed. After a quick jaunt through their course schedules, I found that they were offering the SharePoint Administration course this week in Denver. Since I have family in Denver, that made it easy to do on a church budget. After some fortunate timing involving Clif showing up online (he’s in Texas for the next few weeks) and getting his approval for my travel expenses, a plan was put together. The original plan was to load up the minivan and whisk away my lovely wife and two adorable daughters for a week, but we weren’t able to secure the time off for her, so I sadly have to go to Plan B, which means I go by myself. On the other hand, driving is an expensive way to go at the standard IRS mileage rate, so I booked myself a flight and a rental car on Friday night and saved the church a few hundred bucks in the process.
Added bonus: I can take a SharePoint test as an MCSE elective, which improves the Bird:Stone ratio considerably. I’ll also probably spend most of the flight seeing where my existing knowledge stands with regards to the certification requirements. I’ve been getting an awful lot of on-the-job learning for that stuff.
I’m discovering that the WiFi in Terminal C is much better than that in Terminal A (as I discovered last February on a Southwest flight). Terminal A’s WiFi is nearly unusable. I have a fallback plan, however… I got me a new Mogul with Phone-As-Modem capability (more on that in a later post).
I’ll post more on SharePoint as well.
Keeping things in perspective…
I was perusing a technical support community on LiveJournal today and ran across an entry that made my jaw drop.
This poor IT worker had been working on building a new laptop for a VP’s admin assistant because her Outlook client was running slowly. Coming from a VP, it was a rush request and he got it ready for an early morning deployment. So far so good, doesn’t hurt to look good in front a VP who doesn’t dish out praise easily or often. About the time he gets the machine ready to go, another employee comes running into his office with a major problem.
A director who is working on a three-month mission to darkest Africa has ended up with a cracked laptop screen, rendering the entire unit unusable. Since they are only two weeks into this three month mission, it’s a little hard to get parts or a tech to them and they are almost SOL. As luck would have it, another team is heading out there for something, and can hand carry a replacement laptop out to them. Here’s the catch though, they’re leaving the building in 20 minutes for the airport. The only computer that’s ready to go is the one he just got done building. The only thing to do is to quickly setup the user’s email and hand the computer to the team, and wish them a good trip.
“I call my manager to make her aware that the laptop will be delayed a couple of hours as I build a new machine to replace the one I just sent out. She’s not happy, but I don’t care really. I know I did the right thing. ‘Dead in the water in a third world country’ trumps ‘Slow Outlook'”
The tech made a snap judgment call that seemed to be the right thing to do to ensure the business keeps running smoothly. All is well and good until office politics kick in and he gets called into a meeting. Whereupon he had to explain to the VP, his AA, the Program Manager, and the newly installed Help Desk Manager why he made that decision. 15 minutes later, he “left the office with a new bodily orifice, and stronger desire to drink.” Seems a little excessively painful for doing the right thing.
Alas, this is all too common in the business world. Ego and a sense of entitlement grab a hold of many senior executives who feel it’s their right to get new hardware out of IT simply because of their position. It made me realize how tremendously blessed I am to work in an organization where this sort of thing is an extreme rarity. Our executive team is very well grounded and humble, and this sort of ego trip just doesn’t happen.
That’s not to say that the executives don’t have the occasional drop-everything-emergency, but they do have the wisdom to discern what really does merit the IT department’s full attention and what can wait for us to get a chance to get around to doing it for them.
It’s those sorts of seemingly insignificant things that make Resurrection an awesome place to work. The positive impact on everyone’s stress levels of not having an executive team that behaves they’re royalty is something I can’t even begin to put a dollar figure on. Added to that is being secure in the knowledge that my manager will back me up unless I’m very obviously in the wrong, in which case I need to suck it up and take my lumps.
I’m sure that’s one of the things that made us one of the best churches to work for serve. It is truly a blessing to be part of this team.
Y2DST Headaches and Microsoft Advil 2007
As most of you in the IT world have figured out, Congress’ well-meaning push to save $30-odd million a year in energy costs has ended up costing the IT industry and the economy considerably more than that in the changeover.
After spending 2 weeks patching systems and trying to make the Exchange calendar rebase tool work (unsuccessfully), I woke up Sunday morning hoping for the best and expecting the worst. As it turned out, all our Windows 2000 systems were unpatched. Apparently it was the Windows 2000 patch that was the one Microsoft wanted $4000 for – luckily they published a workaround late last week that involved patching the registry with the new timezone data.
In trying to troubleshoot this further, I’d discovered that Microsoft had set up a DST Support chatroom for anyone suffering from DST pains. I posted my problem and hunkered down to wat, as it looked like the Microsoft experts in the room were pretty buy, mostly with fairly inane questions from people who could have figured out the answer with a quick Google search a few weeks ago. After a short wait, I got a private chat request from JamesC saying he thought he could help me and would be happy to do so in a one-on-one chat. He also helped me boil down KB914387 to the simple language of “Back up your registry, patch it, run this script, and this is the backout procedure”, which allowed me to patch the Windows 2000 servers in fairly short order.
The big issue we’d been running into is that the rebasing tool kept crashing on the third mailbox when trying to generate its list of mailboxes and their associated time zones. I decided to head into the office where I had more screen space as well as some peace and quiet from the kids. As I got in, James sent me a debugger tool to install while he quickly grabbed a bite to eat.
I fired up the Microsoft-provided VM and loaded up the debugger while James ate and set up a LiveMeeting session so he could remotely control the session. This is where it got crazy. James was working from home, on his Mac, via VPN and RDP to his machine at work, from there was connecting to the LiveMeeting server, where he had remote control of my RDP session into the virtual machine running on my laptop, and from that session we frequently had another RDP session going to the mail server. I’m amazed it worked at all. I’m also amazed that Microsoft lets James have a Mac.
James spent the next several hours poring over debugger output and fiddling with assembly code trying to make the application do his bidding. Backing him up was none other than the guy who wrote the rebasing tool in the first place. After several hours of this, they both threw their hands up and resigned themselves to the fact that this approach wasn’t going to work (it must have been a weird issue if the guy who wrote it couldn’t even grok it) and that we’d have to try the manual approach. After a few false starts, James got the tool to do its thing.
Naturally, there are a few users this morning who have some appointments that are “pooched”, but that was expected.
It was truly impressive to watch James engage in Extreme Nerd Sports and poke at assembly and debug code in an effort to make the machine do his bidding. Over the course of the 8 hours, I got to know James a little. I ran across his blog (which I won’t link here to keep his personal blog from being associated with Microsoft), and discovered that he is also a committed Christian – yay for God putting the right people in the right places!
James has been spending the last few months working 16 hours a day on DST conversion. His official job at Microsoft is debugging Exchange code, and from the looks of it, he’s darned good at it. In order to help us poor customers, he’s had to miss not only Valentine’s Day but his wife’s birthday as well. I hope Microsoft makes it worth the trouble. You’ve earned yourself a serious break. Mrs. C, you are a saint. Thank you for letting us pick your hubby’s brain.
The importance of a good disaster recovery plan…
Yesterday, a building in the Waldo neighbourhood of Kansas City burned to the ground. The building housed, among other things, a cafe, a bar, and a bridal shop.
By mid-afternoon, the owner of the bridal shop was on the phone to her customers that were getting married this weekend, as well as to designers in order to get replacement dresses overnighted so as not to impact the weddings taking place. She also said that they were buying new computers this weekend and would start up again soon in a yet-to-be-determined location.
In addition to the incredible level of customer service, it would appear that the owner of this shop had a DR plan in place and executed it. When the worst-case scenario actually happened, she knew exactly what needed to be done to continue operations. If she had good backups, she should have her computers up and running by monday. All she needs is some retail space and to replenish her inventory, and she’s back in business.
There are a few key points here:
- Make sure you have recent and good backups, and that they’re stored somewhere safe. Having them sit next to your computer isn’t going to do you a lick of good if the building crashes down around it in a sea of flames.
- Make sure your DR plan is kept current, that key staff know about it, and that a written copy of it is kept off-site. Periodic disaster recovery drills don’t hurt either.
- If you deal primarily with customers, they need to be taken into consideration with DR planning. Taking care of them even when disaster strikes will pay off huge dividends in the long term.
What not to do:
- Keep your backups onsite (or have one copy onsite and one offsite. The most current one should be the offsite copy). Note that fire safes are meant to keep paper documents safe. In a raging fire, the inside of a fire safe will hit 350 degrees. While it won’t cause the paper to burst into flame, your backup tapes/CD/DVD/whatever have long since melted into a puddle of plastic slag.
- Hope that your backups work. Test them every now and then.