Live Streaming on a Budget: Post Roundup

November 18, 2009 |   |  6 Comments  |  Internet Campus, streaming, Wowza

Since there’s been a flurry of interest lately on my series of posts on live church streaming, here’s an index to the entire set of posts:

Tangentially Related:

For the benefit of the less technically inclined, I’ve moved scripts and code to their own section and updated posts that contained the code with links to the appropriate page.

 

Live Streaming on a Budget (Part eleventy point one) – Metrics Revisited

November 18, 2009 |   |  3 Comments  |  Internet Campus, streaming, Wowza

Back in July, I made a post about metrics and a cheesy VB Script that got the job done, but wasn’t particularly elegant. I’ve since improved on this due to load balancing (I posted about that in September).

I’ve since then learned a bunch about RRDTool, and have put together a script that pulls the XML data, groks it, and then populates an RRD. The net result is that I get a graph like this:

streams-200911151215

This graph gives me the following information: The iPhone stream count (with a 10:1 vertical exaggeration), the Flash stream count, and the total viewer count, which is the sum of Flash and iPhone streams, multiplied by a factor of 1.7 (which we’ve found reasonably reflects how many actual people are watching, versus streams. Then the vertical red line shows the time the peak occurred, and the horizontal line shows the level of that peak. The actual peak numbers are listed on the bottom. The RRD and the graph are set up to take into account Windows streams, just as soon as I find a good way to pull that data from WMI via perl.

The general operation is as follows: There’s a script that’s started in a cron job 10 minutes after the servers are spun up, and it polls the origin server every 10 seconds for its counts and populates the RRD. There’s another cron job that runs every minute to generate a current graph, which is then displayed on a page with some javascript that refreshes the image in realtime. Then, at 12:15 and 6:30, there’s another cron job that takes a snapshot of the previous two hours, puts it into a web-accessible directory, and appends an HTML file with a link for easy access later.

All of the metrics scripts, automation scripts, and graphing tools live on a linux virtual machine that runs on our central campus.

This is big improvement over dumping a vbscript into a CSV and then graphing manually with Excel. This happens automatically, in real time, without me being there.

MIDI Surfaces: Behringer BCF2000

November 16, 2009 |   |  Comment  |  Hardware, Internet Campus, streaming

I ran up to Musician’s Friend this afternoon with C so that my wife could have some peace and quiet to work on seminary and candidacy stuff, and picked up a Behringer BCF2000. It’s so very handy to have the MF outlet locally (their distribution warehouse is up near Liberty, MO), and it’s unfortunate that they’re closing the outlet center at the end of the year.

Unboxing

The BCF2000 is a substantial unit with some heft. The box contains the unit, a 6′ USB cable, 6′ power cord (no lumps or wall warts!) and a documentation pack containing a set of manuals in a number of languages, a catalog, and a sticker. The controller is roughly the size and weight of one of their 8-channel baby mixers. Unit is reasonably well  built and heavy enough that it’s not going to unintentionally wander off the desk

Using it

The documentation is pretty straightforward, considering the plethora of operating modes this device provides for routing MIDI signals. While this unit isn’t nearly as easy to program as the Korg Nano (which uses a GUI that writes programming changes to the unit), it doesn’Behringer BCF2000t rely on any external software to do its thing. The 8 rotary encoders at the top are all you need, and once you get used to it, it’s pretty simple. ETA: Behringer does provide a Java app that lets you do visal programming from the desktop. Very cool.

The B-Control also has the technical yumminess of motorized faders and presets, which make scene changes easy.

Speaking of scenes, one of the big downsides to the unit is that 8 faders is all you get. The only controls that have scene capability are the rotary encoders at the top, which can have up to four scenes (helpful for EQ settings), but no such luck on the faders. The Korg Nano would do multiple fader scenes quite easily. On the other hand, you can gang a bunch of these together through standard MIDI connections.

ETA: I stand corrected. The presets on the BCF are not just for fader positions, but for programming as well. There are several of these.

Random cool tool: MIDI Sniffer – allows you to see what’s coming across the wire.

Tomorrow, we’ll see how well it plays with the VT5 machine.

MIDI Surfaces: Korg NanoKONTROL

November 12, 2009 |   |  Comment  |  Hardware, Internet Campus, streaming

I got the OK from Clif to get the VT5 MIDI interface from Dhomas, and a control surface. The first one to try, simply by virtue of its ready availability at the local Guitar Center was the Korg NanoKONTROL.

This is a USB MIDI device in a plastic shell that’s meant to look suspiciously like one of its parents was a white MacBook. The device offers 9 sets of a fader, a knob, and two lighted buttons, as well as a 6-button set of transport controls and a scene selection button that lets you cycle through 4 different scene presets. It definitely doesn’t havenanoKONTROL_top the build quality of the Mac. For sixty bucks, you can’t expect much, though. Faders, knobs and buttons feel cheap. No software is included with the device, with Korg directing customers to their website to download a driver (optional, it works with the standard Windows USB MIDI driver, but the Korg driver offers some additional functionality. For the people who actually use MIDI for, you know, MUSIC, you’ll be happy to know that this device has two siblings, one with a set of pads, and the other is a 2-octave keyboard – all three are available in black, if you don’t like the Apple Fanboy shade of white.

Programming the unit requires Korg’s software, the Korg Kontrol Editor. It presents a UI that is more than a little reminiscent of a mac (this is aimed at music people, after all) that lets you set the parameters for each control on the unit. As of this post, the software is in version 1.0, and is only able to send CC and MMC commands, and there’s no option for any PC commands. Given that several of the items I want to control on the VT5 require PC commands to change, I find this to be a major shortcoming. Buttons can be set to toggle on/off or be momentary, with attack/decay controls along with what values are represented by the on and off states of the buttons. Similarly, the fader settings allow you to define values for top and bottom, allowing you to reverse the operation of the faders.

On the VT5 side, I downloaded the demo version of MIDI-VT, which allows only the control of the output faders on the VT5’s software audio mixer, but I was able to configure the NanoKontrol unit with very little difficulty, and controls on-screen are very responsive to the fader inputs on the external device. It’s considerably easier than using the mouse. Unfortunately, MIDI-VT doesn’t currently support MMC commands for DDR transport operation.

I contacted Korg about the PC issue, and they responded “As a product that is only designed to be a MIDI controller, it wouldn’t be used nor is it intended for system control that would ordinarily be handled via mouse and keyboard”. Sorry, Korg, that’s not gonna cut it. I want to use this precisely to AVOID using mouse/keyboard controls. Guitar Center, you can have it back.

Overall, this is a great inexpensive solution for an audio mixer control surface in VT5, but Korg’s lack of support for PC commands on the unit severely limits its usefulness for anything beyond the audio mixer.

MIDI Control Surfaces

November 12, 2009 |   |  3 Comments  |  Hardware, Internet Campus, streaming

This post is mostly for my own reference, but putting it out there for anyone else looking for MIDI control surfaces that can be used with VT5 by way of dhomas’ MIDI-VT software. Much discussion on the NewTek forums here. Primary objective is physical control (as opposed to on-screen) of the VT5’s internal audio mixer, with secondary objectives being able to easily control some remote camera parameters like iris/gain/shutter/focus as well as DDR and capture transport controls.

Here’s what I was able to round up so far — I’m starting to understand why keyboard people are gadget freaks (and often broke):

PreSonus FaderPort, $130. Looks like this would be a great controller for both DDR and Capture modules. Despite not having any MIDI ports, this is in fact a MIDI device, which presents itself to your computer as an independent USB MIDI controller. This could potentially pose a problem if you start getting a lot of these. Price is very attractive.

Novation Zero SL MkII, $400-$500. This one just looks cool. Lots of great feedback features, but I’m not sure the plugin can take advantage of them. Good blend of buttons, knobs, and faders.

Akai APC40, $400-$600. Holy buttons! I want one of these just to program the blinkenbuttons. I’m sure I could find a use for all those buttons.Did I mention buttons?

Korg NanoKontrol, $60. Not a lot of buttons on these, but they’re dirt cheap and you can have a bunch of them for different stuff.

Behringer BCF2000, $150. This offers huge bang for the buck, and the VT5 people love it. It also has a cousin with lots of knobs, the BCR2000. Offers USB MIDI controller as well as traditional MIDI ports.

Evolution UC-33, $150-$200. Can’t seem to get much manufacturer information on this one, wondering if it’s discontinued. Looks similar to the Behringer.

Livid Ohm64, $600. This is just a thing of beauty. Buttons and faders and blue blinkies, oh my!

Any others that you MIDI freaks out there have used and like?

Live Streaming on a budget: Wowza Load Balancing (Part … um.. how many is it now?)

September 27, 2009 |   |  Comment  |  Internet Campus, streaming

This week, I discovered that the new version (4.5) of JW Player now supports the Wowza RTMP redirect method of load balancing. Huge props to Richard Lanham for making that possible. Richard is a tremendous asset to the Wowza community.

A few months back, the fine folks at Wowza added a load balancing method where the edge repeaters communicate with the origin server with information about the number of connected clients. In an RTMP redirect, the origin server has a special publishing point (an “application” in the Wowza vernacular) called /redirect that will then tell the player which of the connected edge servers has the lowest load. Charlie Good from Wowza has a post in the Wowza forums that documents how to make this work

The basic mechanism of RTMP redirects for load balancing on Wowza lends itself very well to a dynamic scaling platform such as EC2. As you may recall, we’ve been using round-robin DNS to scale out our streaming, with Wowza’s edge/repeater configuration.

What this means for us is that we can now spin up two repeaters and an origin, and we don’t need to assign an ElasticIP to the repeaters and no longer need to keep a pool of addresses in reserve (at a penny an hour when they’re not connected). If the load gets high on the existing repeaters, I can launch an additional one and the load balancing will send new connections to that server until it’s reached the level of the others.

This makes scaling our streaming much more dynamic than three servers, and is theoretically limitless (within the performance limitations of the origin server). If we get a huge spike for something like the health care forum recently, we can adapt much quicker. It’s also possible to round-robin a pair of origin servers and segment your repeater farms, or use the pair of origin servers for redundancy.

Another feature of the load balancing is a module on the origin server called serverInfoXML that returns statistics on each of the connected repeaters. I’ve hacked together some ugly perl to parse this into some meaningful stats, and it dynamically changes along with the number of servers. The only catch is that it doesn’t work with less than two repeaters due to array dimensioning. If anyone with mad perl skills wants to help make this prettier, and feed the data into a graphing tool like RRDTool, I’d love to hear from you.

Footnote on Automating EC2

July 13, 2009 |   |  Comment  |  Internet Campus, streaming

Yesterday was the first live test of our EC2 automation, and it worked beautifully. One thing I discovered was that when the Wowza servers shut down, Flash Media Encoder simply attempts to reconnect to the server until it succeeds. This is very helpful, since I can just leave the VT5 system on a loop or on the program feed, and as soon as the servers are ready, they’ll start broadcasting.

That way, if something happens and nobody shows up, we’ll at least have something going on the stream.

If you’re using an unattended encoder, you can also take advantage of this (and even schedule start/stop of the encoder software, but it doesn’t have to be timed exactly to the server start/stop.

Live Streaming on a Budget (Part eleventy) – Metrics

July 12, 2009 |   |  Comment  |  Internet Campus, streaming

It’s all well and good that we’re putting this stream out there… But if we’re not reaching anyone, it’s kind of a big waste of money and time. How do we go about finding out how many people we’re reaching?

Fortunately, Wowza has a built-in mechanism for reporting the number of active stream connections via an HTTP connection to the streaming server. It does this primarily for load balancing purposes but the data is easily parsed for other things as well.

I currently have a VB Script (the code is horrid, because I suck at VB)  that connects to each of the streaming servers, parses the response into a numerical value, adds them up to get a total stream count. The script runs on a 5-second delay loop, keeps track of the peak, and gives me the following output, where the red area is the origin server, green is the repeaters, and blue is the iPhone streams. Windows streams are gleaned directly from Windows Media Services.

Stream count output

Stream count output

That’s all well and good, but how many actual people are watching? We know there are several people who watch this alone, but others do it in groups or with their family. Initially, when we benchmarked other churches, we were told that a ratio of about 1.8 people per stream was a pretty reliable guess. We went with that for a while, as we gathered our own data.

To gather our data, we created a sign-in/feedback form for our web audience that functions very much like the friendship pad we pass around in our physical worship services. One key question that you find only online is “how many people are worshipping today?” Based on the aggregated data from these forms (we’ve collected nearly 8,000), we found that our ratio was closer to 1.7, so we started using that for the purposes of reporting. We typically see about 40% of our peak stream count send us a feedback form, so the per-stream count is probably a fairly representative sample. Periodically, we’ll see the ratio jump up to 2:1 in special circumstances such as inclement weather at our central campus, and we’ll adjust the numbers for that service accordingly.

I’m sure there’s a better way than my cheesy vb script, but it works for now.

Live Streaming on a Budget (Intermission) Speedlinks!

July 8, 2009 |   |  Comment  |  Internet Campus, speedlinking, streaming

Here’s a bunch of useful links accumulated recently during the course of my research. They’re here as much for my own recall as they are for your benefit.

Live Streaming on a Budget (Part 5) – Automating EC2

July 8, 2009 |   |  5 Comments  |  Internet Campus, streaming

I mentioned a few posts back that I was looking for a way to automate startup and shutdown of the servers. Thanks to some great sleuthing by Justin Moore at Granger Community Church, I got some scripts to start from. I had to make some modifications to suit our exact purposes, but that was relatively easy.

Ingredients:

Linux system with:

  • Java Runtime (in Ubuntu, the package is sun-java6-jre)
  • Amazon EC2 API Tools (installed in /opt/ec2/tools)
  • Wowza Startup Packages (installed in /root/ec2)
  • EC2 keys (installed in /root/ec2)

Note: Because these scripts are run from cron, you’ll need to put all your environment variables to run EC2 at the beginning of each one.

I have 6 separate versions of the startup and termination scripts, one for each server I need to start. I could roll it into one big script, but putting them in their own individual ones not only lets me do an individual machine manually, I can run them all in parallel from cron, which shortens the startup time.

The startup script functions as follows:

  1. Assign environment variables for EC2 and for the machine parameters
  2. Launch machine with ec2-run-instances, redirect output to a temporary file*
  3. Parse temporary file and extract the instance ID, and put it into an environment variable
  4. Write contents of instance ID environment variable to a file in /root/ec2 for use by the shutdown script
  5. Wait 30 seconds
  6. Start checking instance status every so we know when it’s running (wait 10 seconds to check again if it’s not)
  7. Attach any EBS volumes (Optional – I don’t currently need this, so it’s commented out)
  8. Assign Elastic IP
  9. Execute any additional commands via ssh (Optional, I don’t have any that need to run)

* The original scripts use /tmp/a, which is fine, but I had to make each script do its own temporary file since all 6 were running simultaneously and I ran into problems with getting the right Instance IDs set.

The shutdown script works like this:

  1. Query AWS for all running instances
  2. Issue EC2 termination call
  3. ???
  4. PROFIT!

Lastly, put it in your crontab:

# m h  dom mon dow   command
15 8 * * 0 /root/start-windows.sh
25 8 * * 0 /root/start-origin.sh
25 8 * * 0 /root/start-iphone.sh
25 8 * * 0 /root/start-repeater1.sh
25 8 * * 0 /root/start-repeater2.sh
25 8 * * 0 /root/start-repeater3.sh

0 19 * * 0 /root/term-windows.sh
0 19 * * 0 /root/term-iphone.sh
0 19 * * 0 /root/term-origin.sh
0 19 * * 0 /root/term-repeater1.sh
0 19 * * 0 /root/term-repeater2.sh
0 19 * * 0 /root/term-repeater3.sh

This starts up all of them (except the Windows instance which needs more time) at 8:25 on Sunday morning and shuts them down at 7 on Sunday evening.  (be sure that if you’re using GMT on your Linux box to take that into account).

If you’re using an encoder that can be started and stopped on a schedule, synchronize your times with this, and you’ll be golden. The Wowza EC2 images take about 60-90 seconds to fully get up and running, and the Windows one takes about 10-15 min. Currently, the Windows server pulls from the encoder via a 1:1 NAT rule, so the WME instance can be running before the EC2 server is going. When EC2 is ready, it simply connects to the encoder and is off and running.