Blockbuster is screwed.

Digital distribution is the future of media. Physical media is dead. Yeah, I know, you’ve heard it but don’t believe it.

Today, Penny preached at Resurrection and showed a clip from The Blind Side. Andrea and I had been meaning to see the movie for a while. Since the kids actually went to bed quietly and early, we figured we’d RedBox it and have a nice movie night at home.

One problem though – when your pastor preaches to a couple thousand people and include a clip, there’s a pretty good chance that you’re going to have a hard time finding said movie anywhere near the church. As a backup plan, I fired up google and searched for a torrent version of it. Within about 30 seconds, it was downloading. I then went to Redbox.com, searched for the movie (got lucky and the local box actually had one!), reserved it online, hopped into the car, drove over to the price chopper and picked up the movie. (in retrospect, it would have been faster to take my bike, but it was warm and VERY humid) . I took less than thirty seconds at the kiosk, and drove home. As I sat down in front of my computer, the download had just completed.

Total time elapsed: 17 minutes. In that time, 700MB had downloaded, and hadn’t even uploaded a complete single block (so don’t worry, MPAA, I didn’t actually share any of it). Since I had the DVD, I watched that instead, and had to confront issues such as cleaning the last renter’s fingerprints off the disc and sitting through commercials on the DVD that I paid to rent. I’ll go on faith that the file I downloaded contained the video, and it was kinda nice having a backup plan in case the disc was unusable. Either way, the content owners did get paid.

The process of reserving and picking up a movie on redbox is insanely easy and quick. And downloading a torrent was even easier. If you’re in the business of physical entertainment media, I hope that you’re trying to figure out your exit from that strategy. Browsing and renting at Blockbuster is a painful and expensive process, and that’s why their days are numbered. Redbox has a good thing going, but looking five to ten years down the line, they should be seeing a world of digitally distributed content, not physical media. Netflix has the right idea, but their streaming catalog could use much improvement.

How can content producers leverage the ease and efficiency of peer-to-peer technologies like BitTorrent? The distributed distribution model is incredibly efficient as several companies have discovered where software distribution is concerned. They need to stop fearing peer-to-peer digital distribution and instead leverage its power.

Looking back…

It was nine years ago.

A routine Monday at the office, slaving for my corporate overlords. Five o’clock rolls around and I can finally quit slacking and go home. I pack up, leave my cube. I get about 10 steps away and my phone rings. Probably a server emergency or something. Why I went back and answered it, I have NO earthly idea, but doing so changed my life.

On the other end was the cute intern from another department. She said, “Why don’t you take my roommate out for dinner? She’s nice, and I need the apartment to myself tonight.” It being a Monday night, I had nothing else to do, and getting a real, live date was a hard thing to pass up. I said I’d do it, and then made my way from Broomfield down to Park Meadows in record time (especially at rush hour, pre-TREX!).

Andrea met me at the mall, and we went over to PF Chang’s where my nervous self talked her poor ears off, and then we went to see Shrek. It was totally awkward, but I apparently didn’t completely creep her out, because we met again the following night. And the night after that. Then we did an afternoon doubleheader date, with the Rockies, followed by Aida.

Six weeks later, I was moving to Kansas City to take a job with a new corporate overlord, and she followed me out here (obviously, she was insane!). We’ve now been married almost 8 years and have two (usually) adorable daughters.

Why Bud Selig Was Right

I don’t often blog about sports. Sports writing is not my thing. There are others way better at it than I. But I’ve had a passion for the game of baseball since I was little and my dad took me to a couple of Expos games every summer (which, looking back, prepared me quite nicely for the perpetual disappointment that comes with being a Royals fan these days)

Yesterday’s quasi-perfect game in Detroit was a heartbreaker for any fan of baseball (especially Tigers fans), and certainly for the two people most closely involved: Tigers pitcher Armando Galarraga and MLB umpire Jim Joyce. Galarraga saw his perfect game evaporate in the split second that Joyce missed the ball beating the runner to first base (which, ironically, was being manned by none other than Galarraga).

Joyce has admitted he blew it. Galarraga has clearly forgiven Joyce for denying him a spot in the baseball record books. Both have handled this with grace and absolute class that are one of the great things about baseball. These two checked their egos and remembered, even in the heat of controversy that this is ultimately just a game.

There were cries from all corners for MLB Commissioner Bud Selig to overturn the call and give Galarraga his perfect game. Today he declined to do so, much to the dismay of those fans. I think Selig did the right thing for the purity of the game. The umpires on the field are the ones making the calls, and they need to know without a doubt that the league, especially the commissioner, has their back. Overturning a call in response to public outcry is bad for the game. We can’t have umpires second-guessing themselves during the game, wondering if every call is going to get appealed up the chain.

Even worse, Selig overturning the call would have set an ugly precedent that would likely ultimately lead to video replays in baseball, and that would be a the greatest tragedy to hit baseball since the strikes and night games at Wrigley. Gone would be the drama of managers going up and arguing a bad call, only to have the umpire stand is ground and stick to his decision. There’s no second-guessing in this game. You may think the umpire’s call sucks, but human fallibility is an essential part of baseball, or any sport for that matter.Never mind that video replay would slow the already quite leisurely pace of the game to a crawl.

Ultimately, Galarraga knows he pitched a perfect game, regardless of what ends up in the history books. Joyce admitted he blew it. But then again, these two have achieved something that they otherwise wouldn’t have if it had been a perfect game just like the other twenty (two of which happened in the last month!!). These two have inadvertently (but symbiotically) achieved a great honor: They’re now going to be a part of baseball and sports trivia and will be talked about by fans for years. What more could a ball player ask for?

Browser-Aware Player

One of the big challenges of streaming to the web is the sheer diversity of devices out there.

This past week, I pushed out some modifications to the player code on our live page that switches the player code based on what the user is connecting with. The genesis of this change was a problem with our change to JW Player Version 5 causing our PlayStation users to no longer be able to watch our video since JW v5 requires Flash 10 and Sony apparently doesn’t care about its customers. After a successful test with the Playstation, I extended the code to provide an HTML5 <VIDEO> tag for our iPhone users (allowing us to clear up some the clutter on the sidebar), as well as MMS and RTSP links around a graphic mimicking the Flash-based player in order to provide a consistent user experience for our Android/WebOS/BlackBerry/WinMo users.

EDIT: The main reason I’m not doing straight HTML5 with Flash fallback (a much more elegant solution) is that we’re sending out VP6 for our flash users and a lower-bandwidth h.264 stream for our mobile users. We’re not currently using h.264 for our flash users because of the poor quality of the h.264 encoder in Flash Media Live Encoder. Once we get a “real encoder“, we’ll send out a single set of h.264 streams and use HTML5 with fallback.

The code is here.

A brief history of Palm

With HP’s announcement today that they arepalmpilot1000 purchasing Palm, the loop is complete:

1992: Palm Computing Founded

1995: Palm Computing acquired by US Robotics

1997: US Robotics acquired by 3Com

1998: Palm Computing founders leave to create Handspring

2000: 3Com spins off Palm, Inc.

2002: Palm spins off PalmOS to PalmSource

2003: Palm merges with HandSpring to create PalmOne

2005: PalmOne acquires full rights to Palm trademark and renames back to Palm; PalmSource acquired by ACCESS

2006: Palm buys PalmOS source back  from ACCESS.

2009: 3Com acquired by HP

2010: Palm acquired by HP

The whole thing makes my head dizzy.

Fixing network Priority in Windows : Win7 Update

A long time ago, I made a post about fixing network priority in Windows, and I found myself having to do the same task again on my new Windows 7 system. The process isn’t quite as easy to find under Windows 7/Vista. Here’s the updated version:

Right-click on your network icon and go to the “Network and Sharing center” (if the “Network” icon is on your desktop, you can also get there by right-clicking and going to properties)

Click on “Change Adapter Settings”

Network Advanced

Press the “Alt” Key to show the menu, and click on “Advanced”, then “Advanced Settings”.

(from here, the process is unchanged)

Move the Wired LAN Connection (By Default, “Local Area Connection”) to the top, followed by the wireless connection. Make sure that any VPN virtual adapters come after these, otherwise the VPN will only use the ones above it. This tends to be problematic if you’re using split tunneling, as it will kill any network connection you have.

Once you’ve applied the settings, open a command prompt and run “nslookup” – it should default to the DNS server for your wired network.

Energy-Efficient Lighting: LED Bulbs

The biggest electrical energy drain in your home is usually lights. It goes without saying that if you can address that area, you’ll be a whole lot happier when the electric bill comes.

To this end, we’ve been replacing the lights in our home, starting with CFL a few years ago.Rather than an all-out replacement campaign, we simply replaced bulbs as they went out, under the premise that those are the ones that are on most of the time, and consequently consume more energy. There are several lights in our house that are incandescent, but are rarely used. As such, changing them is not a cost-effective proposition, given the higher cost of the bulbs. There are other areas in the house that are on dimmers, so CFL was not an option there.

One area where I absolutely love the CFL’s usually annoying habit of having to warm up to full brightness over a few minutes? The master bathroom. The overhead light there needs to be reasonably bright, but first thing in the morning, a 100W bulb is an assault on the senses. The 13W CFL there now warms up gently (slower on colder mornings) and isn’t quite so rough when you wake up.

Even CFL bulbs have a limited lifetime (about 2-3x that of incandescents), so those are starting to die now, and need to be replaced. Disposal of fluorescent lights is problematic due to a number of toxic substances involved. Over the past few years, LED bulbs have become a much more mature technology, and the price has come down substantially. Most residential LED bulbs are expected to last 20 years.

Where I’m using LED bulbs in my house:

Kitchen Track Lighting. Gradually replacing the eight halogen GU10 bulbs. So far, I’ve got three 4W Philips LED bulbs from Home Depot ($30). They’re just as bright as the 50W halogens, and they don’t emit any IR to speak of (the halogens would melt butter on the counter!). Net Cost savings over the 10-year lifetime of eight bulbs: around $1500.

LED StackMaster Bedroom. Just got a set of four Dimmable 8W LED lamps from LED Liquidators (please ignore the horrible web design – it seems to be a common problem with online LED retailers) that claim to be equivLED Stack Bulb in Ceiling fanalent to 60W. These go in the ceiling fan and are ideally suited to that application – they’re on a lot, on a dimmer, and the fan has a lot of vibration, which is really hard on incandescents. Net cost savings over 20-year lifetime of the bulbs: $800.

And if you’re into that sort of thing, it also reduces your carbon footprint.

Update, December 2010: All four of these bulbs have failed – they’ve got entire strips of LEDs that are flickering or flat out not working. An e-mail to customer service in October went unanswered. Called by phone and they said “customer service will call back and take care of it”. We shall see.

Anatomy of an online worship service

(or, How Amazon Cloudwatch helps manage Wowza server load)02-21-10-AM-AWS

This morning I woke up to two things: Beautiful Kansas City February weather (aka, an ice storm), and a voicemail from the Senior Pastor, asking if we had sufficient online capacity to support a larger-than-usual stream audience. Online worship streaming is a great option for these weather events that have been so common this winter (and not just in the KC area – we see increased online attendance when the weather gets foul elsewhere, like the DC storms of a few weeks ago).

My first indication that this was going to be a big event was Woopra showing 30 people on the web page half an hour before we start sending any kind of video (which is itself 75 minutes before we actually start the morning service). Usually there are two or three. Fifteen minutes after we started sending video, we were already cranking out 20-30 streams (again, we usually only have a small handful at this point).

02-21-10-AM-CPU

AWS CPU Usage

Most weeks, we run two Wowza repeaters pulling from a single origin server, which gives us plenty of capacity. I had to spin up a third repeater by the beginning of the pre-service music, a fourth about 10 minutes later, and a fifth after five more minutes. I set my threshold for spinning a new server at 75% CPU on the repeaters, as indicated by the AWS CloudWatch monitors. In the case of a heavy influx of viewers, this gives the new instance enough time to get up and running before the other repeaters hit 100% CPU.  Wowza tells me this is at about 180Mbit/sec on a small instance, which for us means around 300 streams. The CPU threshold of 75% works out to about 260 streams.

Unfortunately for our online worshipers, our web server was bogging down pretty hard at

Web Server CPU usage

Web Server CPU usage

the beginning of the service, where the two CPU cores were maxed out for about 15-20 minutes, which translated into slower page loads. The database server wasn’t sweating too hard, so I suspect this could have been helped with better PHP caching. Fortunately for me, this had the effect of slowing down the rate of incoming streams, which allowed me to get new repeaters going before the existing ones started choking.

You can see in the graph where we added new repeaters, and how fast they ramped up. It also shows how incredibly well Wowza’s built-in load balancing works. We eventually leveled out at a little over 1100 streams, which meant our EC2 instances were cranking out 600-700 Mbps for nearly an hour:

AWS Network Usage

AWS Network Usage

Meanwhile, this is what we were seeing on Woopra (note the fortunate souls escaping the ice storm in Aruba and the Cayman Islands!):

2-21-10-AM

Next step is to define rules in Cloudwatch for automatically scaling. For that to work, I’m going to need to build my own Wowza AMI, since the current method of starting repeaters involves sending the instance a startup package from the client. I’ll need to build this configuration into the server for CloudWatch scaling to work properly.

VMWare opens academic pricing to non-profits!

This just in from my Dell rep, and confirmed by several in the CITRT Twitterverse: VMWare has opened up their academic pricing to non-profits. Jason Powell has the pricing information at his blog. The relevant bits from VMWare:

Definition and Requirements of a Non-Profit Entity for Eligibility to Participate in the VMware Academic Buying Program:
The following U.S. Non-Profits under U.S. Tax Code 501(c) are eligible to participate in the VMware Academic Buying Program.

  • Non-Profits are defined under U.S.Tax Code 501(c)(3) as an organization which is organized and operated exclusively for one of the following purposes:
    • Religious
    • Charitable
    • Scientific
    • Literary
    • Educational
    • In the prevention of cruelty to children or animals
    • In the testing for public safety
    • Fostering national or international amateur sports competition
  • Non-Profits under U.S. Tax Code 501(d) Religious or Apostolic Organizations
  • Non-Profits under U.S. Tax Code 501(e) Cooperative Health Services provided to Hospitals
  • Non-Profits under U.S. Tax Code 501(f) Cooperative Service Organizations of Operating Educational Organizations providing cooperative investment services for Educational Organizations
  • Non-Profits under U.S. Tax Code 501(k) Child Care Organizations which supply child care to children with working parents
  • Non-Profits under U.S. Tax Code 501(n) Charitable Risk Pools which pools insurance risks of 501(c) (3) Organizations

Not all nonprofits qualify, of course:

U.S. Organizations not eligible to participate in the VMware Academic Buying Program:

  • U.S. Tax Code 501(c) 1 Non-Profits as defined as Corporate Organization under Acts of Congress, or as Instrumentalities of the United States.
  • U.S. Tax Code 501(c) 2 Non-Profits as defined as Title Holding Corporations for Exempt Organizations, or those who hold title to property owned by Exempt Organizations.
  • Organizations filed under 501(c) (4 through 27). This includes:
    • Teachers’ Retirement Fund Associations
    • Domestic Fraternal Societies and Associations
    • Fraternal Beneficiary Societies and Associations
    • Social and Recreation Clubs
    • Business Leagues, Chambers of Commerce, Real Estate Boards, etc.
    • Labor, Agricultural, and Horticultural Organizations
    • Civic Leagues, Social Welfare Organizations, and Local Associations of Employees
    • Political organizations
    • Labor or fraternal organizations
  • Other Organizations not eligible:
    • Organizations that are an integral part of local government or have governmental powers
    • Hospitals not wholly owned by a University
    • Health Management organizations (HM)
    • Preferred Provider organization (PPO)
    • Non-profits that are not charitable organizations or act as non-profit lobbying groups
    • Private Foundations
    • Academic facilities that qualify for Academic status.

If you’re not in the US, same sort of deal applies, if your organization is equivalently defined as non-profit under your local tax codes. Contact your VMWare reseller for details.

Why consumer IT support sucks

laptop-destroy

Photo: bdtyre

Working in an IT shop, sometimes you have the luxury of forgetting just how bad consumer-grade hardware support has gotten. I’ve been absolutely spoiled by our Dell team, so the past few weeks have been a serious reality check.

About a year and a half ago, I bought a Toshiba laptop for my wife after her desktop computer’s power supply had unleashed its magic smoke. She was needing an upgrade, and we opted for the laptop since she was going to be starting school. After shopping arond, we found a good deal at Costco which included a color laser printer. Added bonus of buying at Costco is that on computers (and TVs) they extend the standard 1-year manufacturer warranty out to two years.

Somewhere around the middle of the fall semester, Andrea noticed the power connector inside the laptop was getting loose. Having encountered this sort of problem before (and I understand it’s actually quite common on consumer laptops), I suspected the plastic shell of the connector had come loose. They typically have three or four plastic pegs that secure the body to the circuit board, taking the mechanical strain off the soldered electrical leads. It’s not uncommon after repeated plug/unplug cycles for one or more of these pegs to break off, causing the connector to come loose. The long-term risk is that metal fatique will then break the electrical connectors. If you’re lucky, it won’t short out in the process. Business-grade laptops usually use much better connectors and methods of securing them. It’s just one of those corners you have to cut to sell $500 laptops to the masses.

Along the way, her touchpad started acting flaky as well, so we figured we’d call Costco and get this fixed while we still had some time left in the warranty. After trying to explain to someone in India for 45 minutes that the problem was not in the external power adapter, I got tired of him sending me on hold to talk to his supervisor, and asked him to transfer me. I spoke with Wade, who was quite helpful, and got the dispatch sent out to their service provider, an outfit in the Orlando area called “Encompass Service Solutions”. They sent me an empty box with packing material via 2-day air. That was two weeks ago.

Since I was sending the computer offsite to some company I’d never heard of, there was not only a high likelihood that they were gonna wipe the system and do a factory reload (crapware and all), there was absolutely no way I was trusting them with the security of our data (even with backups, I’d rather they not have access to my banking data, thankyouverymuch). So I picked up a blank hard drive at Micro Center and shipped it off with the blank.

A week later, it comes back, with the technician notes that the system was cleaned and the CPU “repacked”, and the OS reloaded. No mention of either of the original problems on the service ticket. Sure enough, the power connector is in even worse shape than it was when we sent it off. I call Costco and gripe. They’re not amused. A little later, I flip it over to put our hard drive back in and discover that someone went crazy with stickers. There are 4 new stickers with barcodes, and “Inspected by” sticker, and half a dozen “Warranty Void if broken or removed” stickers over the chassis screws, as well as those to the hard drive and memory bays. I’m a geek, so I’m genetically predisposed to ignore such stickers, and I carefully removed the ones on the drive and memory bays, and put our hard drive back in.

Then things got interesting. I power the machine on, and the display remains dark. No backlight, no image, nothing. I check das blinkenlights, and they’re behaving like a normal boot process. After some fumbling and troubleshooting, I hang an external monitor from it and fire it up again. That part works. It seems that the people in QA at Encompass were sleeping that day, because I’d imagine that a non-functional display normally would cause QA to kick it back to the tech who forgot to plug in the ribbin cable feeding the display. Apparently at Encompass, this is not the case.

So now we have a laptop that is effectively a compact desktop. A couple calls back and forth to Costco, and Encompass wants me to send it back to them for repair. Only here’s the problem, guys, I don’t trust you to not break it worse this time, and Andrea needs her laptop this week, that’s why we sent it to you when we did. Even better, you can’t e-mail me a UPS label for me to use the existing box, you insist on sending me another empty box via UPS. Not only is that insanely wasteful of diesel, jet fuel, and cardboard, and it adds 3 days to the process. Andrea can’t be without the laptop any longer than she already has. She has interviews this week and school starts up next week. I don’t care how fast you expedite it, you’ve proven already that you half-ass the job when it’s a rush. If I wanted that kind of shoddy work done on my laptop, I would have handed it off the the Geek Squad at a local Best Buy.

Wade, the guy at Costco, is trying to come up with alternatives. At this point, acceptable outcomes are that we get an onsite tech to do a motherboard replacement, find me a local depot in the Kansas City area, or you give us at least partial credit toward a replacement system at the warehouse.

When you buy a laptop, spring for the onsite support. If anything happens, it will save your sanity. Next time, I’m buying a Dell.

I’d love to hear your depot repair horror stories.