Amazon and new Cloudfront Streaming

December 18, 2009 |   |  Comment  |  streaming

Earlier this week, I got an e-mail from Amazon Web Services, with some new goodies being announced.

The first was new pricing for Wowza. While the cost of each instance-hour is going up slightly, the cost of bandwidth is dropping about 15%. This makes me happy.

The second was that Wowza has released version 2. I’ve been working with preview versions of this since last summer with our iPhone streaming. They’ve made some very cool improvements to the product.

Lastly was an item that intrigued me. Amazon has, via Flash Media Server, added RTMP streaming capability to Cloudfront, Amazon’s cloud answer to CDN. Now instead of being able to widely distribute files, you have the capability of setting up a distribution that provides RTMP streaming of any supported video file in an S3 bucket. If you use S3 for on-demand video content, this is big for you. No longer do your viewers have to download the entire file (and run up the bandwidth meter in the process). They can now skip directly to the points they need and only use the bandwidth for stuff they actually watch.

This is potentially very good news for services that serve on-demand content from S3 (such as blip.tv)

It’s not so good news for the folks at Wowza, because I no longer have to spin up a Wowza instance to serve content stored in S3. Luckily for Wowza, the Cloudfront streaming doesn’t do live video.

It’ll be interesting to see how this plays out. As it currently stands, we could replace blip.tv with this functionality, but for the small cost, we get a whole lot of value from Blip.

On the perils of LED lighting and video…

December 10, 2009 |   |  Comment  |  streaming

For the last couple of years, we’ve used LED Christmas lights in our sanctuary. Considering how many we have (hundreds), the electricity savings are probably non-trivial.

All our LED strings in the sanctuary are plugged into either a stage dimmer or, where a dimmer port was unavailable, an Elation UniBar hooked into an RC4 Magic wireless DMX receiver (with the transmitter wired into DMX up in the catwalks). This allows us to control the Christmas lights along with the rest of the theatrical lighting via the Hog. It’s a very nice setup.

The other day, when Frank was running the stream, he saw the Christmas lights were fading in and out in sequence, and called up to the Penalty Box (the plexiglass-wrapped area at the back of house where the lighting operator and worship producer sit) and asked them to quit playing with the lights. As it turns out, they weren’t and the lights were all on. Mysteriously, they were fading in and out in sequence on the wide shot camera. When we looked at them on one of the other remote cameras, everything looked normal.

Then it hit me. I went to the remote control on the wide camera and cranked down the shutter speed, and lo, the lights gradually came together until they were all on. This is what it looked like:

Most stage dimmers operate by switching the AC cycle on and off via pulse width modulation. LEDs then only show one half of the AC sine wave, making them strobe, effectively reproducing the square-wave pulses that are modulating the dimmers. What We were seeing on the cameras was a beat frequency of the camera’s shutter speed and the strobing of the LEDs. You don’t see this on incandescent lighting because of the thermal persistence of the filaments. But why were the lights cycling at different times? Each one was connected to a different dimmer circuit, and those circuits are spread among the three AC phases coming into the dimmer room (which has a monster 2000-amp breaker).

So, if you’re shooting video of anything that has LED lights in it, make sure your shutter speed is at 1/60, or the lights are going to start acting strangely.

Live Streaming on a Budget: Post Roundup

November 18, 2009 |   |  6 Comments  |  Internet Campus, streaming, Wowza

Since there’s been a flurry of interest lately on my series of posts on live church streaming, here’s an index to the entire set of posts:

Tangentially Related:

For the benefit of the less technically inclined, I’ve moved scripts and code to their own section and updated posts that contained the code with links to the appropriate page.

 

Live Streaming on a Budget (Part eleventy point one) – Metrics Revisited

November 18, 2009 |   |  3 Comments  |  Internet Campus, streaming, Wowza

Back in July, I made a post about metrics and a cheesy VB Script that got the job done, but wasn’t particularly elegant. I’ve since improved on this due to load balancing (I posted about that in September).

I’ve since then learned a bunch about RRDTool, and have put together a script that pulls the XML data, groks it, and then populates an RRD. The net result is that I get a graph like this:

streams-200911151215

This graph gives me the following information: The iPhone stream count (with a 10:1 vertical exaggeration), the Flash stream count, and the total viewer count, which is the sum of Flash and iPhone streams, multiplied by a factor of 1.7 (which we’ve found reasonably reflects how many actual people are watching, versus streams. Then the vertical red line shows the time the peak occurred, and the horizontal line shows the level of that peak. The actual peak numbers are listed on the bottom. The RRD and the graph are set up to take into account Windows streams, just as soon as I find a good way to pull that data from WMI via perl.

The general operation is as follows: There’s a script that’s started in a cron job 10 minutes after the servers are spun up, and it polls the origin server every 10 seconds for its counts and populates the RRD. There’s another cron job that runs every minute to generate a current graph, which is then displayed on a page with some javascript that refreshes the image in realtime. Then, at 12:15 and 6:30, there’s another cron job that takes a snapshot of the previous two hours, puts it into a web-accessible directory, and appends an HTML file with a link for easy access later.

All of the metrics scripts, automation scripts, and graphing tools live on a linux virtual machine that runs on our central campus.

This is big improvement over dumping a vbscript into a CSV and then graphing manually with Excel. This happens automatically, in real time, without me being there.

MIDI Surfaces: Behringer BCF2000

November 16, 2009 |   |  Comment  |  Hardware, Internet Campus, streaming

I ran up to Musician’s Friend this afternoon with C so that my wife could have some peace and quiet to work on seminary and candidacy stuff, and picked up a Behringer BCF2000. It’s so very handy to have the MF outlet locally (their distribution warehouse is up near Liberty, MO), and it’s unfortunate that they’re closing the outlet center at the end of the year.

Unboxing

The BCF2000 is a substantial unit with some heft. The box contains the unit, a 6′ USB cable, 6′ power cord (no lumps or wall warts!) and a documentation pack containing a set of manuals in a number of languages, a catalog, and a sticker. The controller is roughly the size and weight of one of their 8-channel baby mixers. Unit is reasonably well  built and heavy enough that it’s not going to unintentionally wander off the desk

Using it

The documentation is pretty straightforward, considering the plethora of operating modes this device provides for routing MIDI signals. While this unit isn’t nearly as easy to program as the Korg Nano (which uses a GUI that writes programming changes to the unit), it doesn’Behringer BCF2000t rely on any external software to do its thing. The 8 rotary encoders at the top are all you need, and once you get used to it, it’s pretty simple. ETA: Behringer does provide a Java app that lets you do visal programming from the desktop. Very cool.

The B-Control also has the technical yumminess of motorized faders and presets, which make scene changes easy.

Speaking of scenes, one of the big downsides to the unit is that 8 faders is all you get. The only controls that have scene capability are the rotary encoders at the top, which can have up to four scenes (helpful for EQ settings), but no such luck on the faders. The Korg Nano would do multiple fader scenes quite easily. On the other hand, you can gang a bunch of these together through standard MIDI connections.

ETA: I stand corrected. The presets on the BCF are not just for fader positions, but for programming as well. There are several of these.

Random cool tool: MIDI Sniffer – allows you to see what’s coming across the wire.

Tomorrow, we’ll see how well it plays with the VT5 machine.

MIDI Surfaces: Korg NanoKONTROL

November 12, 2009 |   |  Comment  |  Hardware, Internet Campus, streaming

I got the OK from Clif to get the VT5 MIDI interface from Dhomas, and a control surface. The first one to try, simply by virtue of its ready availability at the local Guitar Center was the Korg NanoKONTROL.

This is a USB MIDI device in a plastic shell that’s meant to look suspiciously like one of its parents was a white MacBook. The device offers 9 sets of a fader, a knob, and two lighted buttons, as well as a 6-button set of transport controls and a scene selection button that lets you cycle through 4 different scene presets. It definitely doesn’t havenanoKONTROL_top the build quality of the Mac. For sixty bucks, you can’t expect much, though. Faders, knobs and buttons feel cheap. No software is included with the device, with Korg directing customers to their website to download a driver (optional, it works with the standard Windows USB MIDI driver, but the Korg driver offers some additional functionality. For the people who actually use MIDI for, you know, MUSIC, you’ll be happy to know that this device has two siblings, one with a set of pads, and the other is a 2-octave keyboard – all three are available in black, if you don’t like the Apple Fanboy shade of white.

Programming the unit requires Korg’s software, the Korg Kontrol Editor. It presents a UI that is more than a little reminiscent of a mac (this is aimed at music people, after all) that lets you set the parameters for each control on the unit. As of this post, the software is in version 1.0, and is only able to send CC and MMC commands, and there’s no option for any PC commands. Given that several of the items I want to control on the VT5 require PC commands to change, I find this to be a major shortcoming. Buttons can be set to toggle on/off or be momentary, with attack/decay controls along with what values are represented by the on and off states of the buttons. Similarly, the fader settings allow you to define values for top and bottom, allowing you to reverse the operation of the faders.

On the VT5 side, I downloaded the demo version of MIDI-VT, which allows only the control of the output faders on the VT5’s software audio mixer, but I was able to configure the NanoKontrol unit with very little difficulty, and controls on-screen are very responsive to the fader inputs on the external device. It’s considerably easier than using the mouse. Unfortunately, MIDI-VT doesn’t currently support MMC commands for DDR transport operation.

I contacted Korg about the PC issue, and they responded “As a product that is only designed to be a MIDI controller, it wouldn’t be used nor is it intended for system control that would ordinarily be handled via mouse and keyboard”. Sorry, Korg, that’s not gonna cut it. I want to use this precisely to AVOID using mouse/keyboard controls. Guitar Center, you can have it back.

Overall, this is a great inexpensive solution for an audio mixer control surface in VT5, but Korg’s lack of support for PC commands on the unit severely limits its usefulness for anything beyond the audio mixer.

MIDI Control Surfaces

November 12, 2009 |   |  3 Comments  |  Hardware, Internet Campus, streaming

This post is mostly for my own reference, but putting it out there for anyone else looking for MIDI control surfaces that can be used with VT5 by way of dhomas’ MIDI-VT software. Much discussion on the NewTek forums here. Primary objective is physical control (as opposed to on-screen) of the VT5’s internal audio mixer, with secondary objectives being able to easily control some remote camera parameters like iris/gain/shutter/focus as well as DDR and capture transport controls.

Here’s what I was able to round up so far — I’m starting to understand why keyboard people are gadget freaks (and often broke):

PreSonus FaderPort, $130. Looks like this would be a great controller for both DDR and Capture modules. Despite not having any MIDI ports, this is in fact a MIDI device, which presents itself to your computer as an independent USB MIDI controller. This could potentially pose a problem if you start getting a lot of these. Price is very attractive.

Novation Zero SL MkII, $400-$500. This one just looks cool. Lots of great feedback features, but I’m not sure the plugin can take advantage of them. Good blend of buttons, knobs, and faders.

Akai APC40, $400-$600. Holy buttons! I want one of these just to program the blinkenbuttons. I’m sure I could find a use for all those buttons.Did I mention buttons?

Korg NanoKontrol, $60. Not a lot of buttons on these, but they’re dirt cheap and you can have a bunch of them for different stuff.

Behringer BCF2000, $150. This offers huge bang for the buck, and the VT5 people love it. It also has a cousin with lots of knobs, the BCR2000. Offers USB MIDI controller as well as traditional MIDI ports.

Evolution UC-33, $150-$200. Can’t seem to get much manufacturer information on this one, wondering if it’s discontinued. Looks similar to the Behringer.

Livid Ohm64, $600. This is just a thing of beauty. Buttons and faders and blue blinkies, oh my!

Any others that you MIDI freaks out there have used and like?

Live Streaming on a budget: Wowza Load Balancing (Part … um.. how many is it now?)

September 27, 2009 |   |  Comment  |  Internet Campus, streaming

This week, I discovered that the new version (4.5) of JW Player now supports the Wowza RTMP redirect method of load balancing. Huge props to Richard Lanham for making that possible. Richard is a tremendous asset to the Wowza community.

A few months back, the fine folks at Wowza added a load balancing method where the edge repeaters communicate with the origin server with information about the number of connected clients. In an RTMP redirect, the origin server has a special publishing point (an “application” in the Wowza vernacular) called /redirect that will then tell the player which of the connected edge servers has the lowest load. Charlie Good from Wowza has a post in the Wowza forums that documents how to make this work

The basic mechanism of RTMP redirects for load balancing on Wowza lends itself very well to a dynamic scaling platform such as EC2. As you may recall, we’ve been using round-robin DNS to scale out our streaming, with Wowza’s edge/repeater configuration.

What this means for us is that we can now spin up two repeaters and an origin, and we don’t need to assign an ElasticIP to the repeaters and no longer need to keep a pool of addresses in reserve (at a penny an hour when they’re not connected). If the load gets high on the existing repeaters, I can launch an additional one and the load balancing will send new connections to that server until it’s reached the level of the others.

This makes scaling our streaming much more dynamic than three servers, and is theoretically limitless (within the performance limitations of the origin server). If we get a huge spike for something like the health care forum recently, we can adapt much quicker. It’s also possible to round-robin a pair of origin servers and segment your repeater farms, or use the pair of origin servers for redundancy.

Another feature of the load balancing is a module on the origin server called serverInfoXML that returns statistics on each of the connected repeaters. I’ve hacked together some ugly perl to parse this into some meaningful stats, and it dynamically changes along with the number of servers. The only catch is that it doesn’t work with less than two repeaters due to array dimensioning. If anyone with mad perl skills wants to help make this prettier, and feed the data into a graphing tool like RRDTool, I’d love to hear from you.

AT&T FAIL!!!!

August 14, 2009 |   |  Comment  |  Grrr!, Wireless

Time Warner’s internet service has been flaky at best at my house lately, despite swapping out the modem. So I figured I’d see if AT&T would give me data-only u-verse. They will, and when I started asking about additional fees, the conversation took a sharp weird turn.

Delilah: Thank you for using AT&T! How can I help you today?
you: looking for info on data service.
you: need 1Mbps or higher upstream, 10Mbps downstream
Delilah: Yes, we offer up to 18 Mbps, if you are not ordering the TV service, then we do have an installation fee.
you: what’s the pricing for 12/1.5 ?
Delilah: There is a learn more blue tab on U-verse page. (note: it’s ORANGE)
Delilah: Please click to view speeds and pricing, the installation is $149.
you: what am I clicking on?
Delilah: click on build your own bundle.
you: all the tabs on the page are orange.
Delilah: Build your own bundle first.
Delilah: Then you will see Uverse internet
you: i just see packages
Delilah: Besides the internet
Delilah: Do you see where it says Max 18
You: so if I were to get 6/1, what would it cost me?
Delilah: The Elite is $35 a month.
You: what else would I be paying? (was asking about fees, this is where it got interesting…)
Delilah: Just the installation fee, this is a wireless service, no modem needed.
you: uh… wireless???
Delilah: Taxes and fees are less than $6
you: I thought this was DSL
you: I don’t think I’ll be able to get an antenna put on my roof
Delilah: Wireless internet
Delilah: It is through the phone line, yes,
Delilah: You do not need one.
you: is it wireless, or does it use DSL over the phone lines?
Delilah: It is brand new technology, (uh-oh… stand by for clueless marketing BS…)
Delilah: A fiber optic line that runs through your phone line. (wait, WHAT???)
you: well, I don’t have fiber running to my home, just copper
Delilah: Did you check on availability?
Delilah: Our technicians run the wire.
Delilah: As long as it is available at your address,,
you: yes, u-verse was built out in my neighborhood 3 years ago, I remember it well, as you guys left a gaping hole in my front yard for 8 months. (it took me several calls to AT&T, before I found someone who cared enough to inform the contractor that they’d forgotten to fill it back up after redoing the splice. You can see it on Bing Maps Bird’s Eye View. The grass still isn’t quite right)
you: but there’s no fiber here.
you: the technicians told me this was DSL-based technology.
Delilah: I am sorry, I am a bit confused, are you wanting just regular DSL?
you: i need internet service. But there’s no fiber running to my house. (and if there was, 18/1 would be pathetically slow)
Delilah: Would you like a link to check on DSL at your home?
Delilah: There is no need for a wire of anykind.
you: if there’s no need for wire of any kind, how does it get to my house without an antenna on the roof?
Delilah: DSL has never needed an antenna.
Delilah: It runs through the phone line.
you: that’s a wire.
Delilah: It is already there.
you: yes, but that’s not “wireless”
Delilah: You get a wireless modem if you like.
Delilah: You can get signals up to 250 ft.
Delilah: There is no antenna at all.
Delilah: No antenna.
Delilah: The days of antennas are over.
Delilah: Fiber optic lines that run through the phone.
you: then how do you get the signal to my house wirelessly?
Delilah: I am sorry you do not  understand, would you like to call in?
Delilah: Through your phone line.
you: then it’s not wireless.
Delilah: You get a wireless  modem.
you: why would I need a wireless modem, since it comes through the phone lines
Delilah: Since I cannot assist you any further, this chat will now close. Thank for chatting with AT&T.
Thank you for chatting with AT&T today. Have a great day.

OK, I know what she meant, but she’s doing a lousy job of selling the product, since she seems to have a tenuous grasp on what this is in the first place. I don’t think I’ll be getting internet from them anytime soon. At least I can get it without TV now… although I think I’d be better off getting the TV service for 3 months instead of the $150 install fee.

Let’s make this clear, AT&T. U-Verse is NOT a fiber-to-the-home service. It’s a high-speed DSL service (54Mbit) that goes to a fiber-fed node in the neighborhood within the wire limits of the DSL Technology. I know Verizon’s cleaning your clock with FTTH right now, but stop misrepresenting copper-based DSL services as fiber. It just ain’t true. By your definition DSL service has been “fiber” for years in every CO that’s fed by an OC link.

Footnote on Automating EC2

July 13, 2009 |   |  Comment  |  Internet Campus, streaming

Yesterday was the first live test of our EC2 automation, and it worked beautifully. One thing I discovered was that when the Wowza servers shut down, Flash Media Encoder simply attempts to reconnect to the server until it succeeds. This is very helpful, since I can just leave the VT5 system on a loop or on the program feed, and as soon as the servers are ready, they’ll start broadcasting.

That way, if something happens and nobody shows up, we’ll at least have something going on the stream.

If you’re using an unattended encoder, you can also take advantage of this (and even schedule start/stop of the encoder software, but it doesn’t have to be timed exactly to the server start/stop.