Multi-tenant Virtual Hosting with Wowza on EC2

That’s a mouthful, isn’t it?

I recently needed to migrate a couple of Wowza Streaming Engine tenants on a baremetal server that was getting long in the tooth, and was getting rather expensive. These tenants were low-volume DVR or HTTP transmuxing customers, with one transcoding customer that required some more CPU power. But this box was idle most of the time. So I decided to move it over to AWS and fire up the box only when necessary. Doing this used to be a cumbersome process with the AWS command-line tools that were Java-based. The current incarnation of tools is quite intuitive and runs in Python, so there’s not a lot of insane configuration and scripting to do.

You may recall my post from a few years back about multi-tenant virtual hosting. I’m going to expand on this and describe how to do it within the Amazon EC2 environment, which has historically limited you to  a single IP address on a system.

The first step to getting multiple network interfaces on EC2 is to create a Virtual Private Cloud (VPC) and start your EC2 instances within your VPC. “Classic” EC2 does not support multiple network interfaces.

Once you’ve started your Wowza instance within your VPC (for purposes of transcoding a single stream, I’m using a c4.2xlarge instance), you then go to the EC2 console, and on the left-hand toolbar, under “network and security” is a link labeled “Network Interfaces”. When you click on that, you have a page listing all your active interfaces.

To add an interface to an instance, simply create a network interface, select the VPC subnet it’s on, and optionally set its IP (the VPC subnet is all yours, in dedicated RFC1918 space, so you can select your IP). Once it’s created, you can then assign that interface to any running instance. It shows up immediately within the instance without needing to reboot.

Since this interface is within the VPC, it doesn’t get an external IP address by default, so you’ll want to assign an ElasticIP to it if you wish to have it available externally (in most cases, that’s the whole point of this exercise)

Once you have the new interface assigned, simply configure the VHosts.xml and associated VHost.xml files to listen to those specific internal IP addresses, and you’re in business.
As for scheduling the instance? On another machine that IS running 24/7 (if you want to stick to the AWS universe, you can do this in a free tier micro instance), set up the AWS command line tools and then make a crontab entry like this:

30 12 * * 1-5 aws ec2 start-instances --instance-ids i-XXXXXXXX
35 12 * * 1-5 aws ec2 associate-address --network-interface-id eni-XXXXXXXX --allocation-id eipalloc-XXXXXXXX
35 12 * * 1-5 aws ec2 associate-address --network-interface-id eni-XXXXXXXX --allocation-id eipalloc-XXXXXXXX
30 15 * * 1-5 aws ec2 stop-instances --instance-ids i-XXXXXXXX 

This fires up the instance at 12:30pm on weekdays, assigns the elastic IPs to the interfaces, and then shuts it all down 3 hours later (because this is an EBS-backed instance in a VPC, stopping the instance doesn’t nuke it like terminating does, so any configuration you make on the system is persistent)

Another way you can use this is to put multiple interfaces on an instance with high networking performance and gain the additional bandwidth of the multiple interfaces (due to Java limitations, there’s no point in going past 4 interfaces in this use case), and then put the IP addresses in either a round-robin DNS or a load balancer, and simply have Wowza bind to all IPs (which it does by default).

Amazon EC2 and DNS

Just discovered an interesting little tidbit about using DNS from within Amazon’s EC2:

When resolving the public DNS name of an EC2 instance, from another EC2 instance, you will get the internal IP address of that instance.This is useful if you have multiple EC2 instances talking to each other.

For example, if you have a Wowza edge/origin setup, you have your origin set up on an elastic IP for consistency in your configuration, and point your edge servers to that IP.

Now this may seem insignificant until you remember that any traffic between EC2 instances via the public IP (elastic IP or not) is going to incur a charge of 1 cent per gigabyte. If you’ve got a large streaming setup, that can add up.

If you want to use your own domain name for your server, be sure to use a CNAME record to the public IP instead of an A record. The A record will always return the public IP. The CNAME will tell the nameserver what the public DNS name is for that instance, which EC2’s nameservers will then return as the internal address.

With an A record to the public (elastic) IP:

ubuntu@ip-10-245-bar-baz:~$ nslookup wms.domain.org.
Non-authoritative answer:
Name:   wms.domain.org
Address: 50.16.XXX.YYY

With a CNAME record to the public DNS name:

ubuntu@ip-10-245-bar-baz:~$ nslookup wms.domain.org.
Non-authoritative answer:
wms2.domain.org      canonical name = ec2-50-16-XXX-YYY.compute-1.amazonaws.com.
Name:   ec2-50-16-XXX-YYY.compute-1.amazonaws.com
Address: 10.114.AAA.BBB

Footnote on Automating EC2

Yesterday was the first live test of our EC2 automation, and it worked beautifully. One thing I discovered was that when the Wowza servers shut down, Flash Media Encoder simply attempts to reconnect to the server until it succeeds. This is very helpful, since I can just leave the VT5 system on a loop or on the program feed, and as soon as the servers are ready, they’ll start broadcasting.

That way, if something happens and nobody shows up, we’ll at least have something going on the stream.

If you’re using an unattended encoder, you can also take advantage of this (and even schedule start/stop of the encoder software, but it doesn’t have to be timed exactly to the server start/stop.

Live Streaming on a Budget (Part eleventy) – Metrics

It’s all well and good that we’re putting this stream out there… But if we’re not reaching anyone, it’s kind of a big waste of money and time. How do we go about finding out how many people we’re reaching?

Fortunately, Wowza has a built-in mechanism for reporting the number of active stream connections via an HTTP connection to the streaming server. It does this primarily for load balancing purposes but the data is easily parsed for other things as well.

I currently have a VB Script (the code is horrid, because I suck at VB)  that connects to each of the streaming servers, parses the response into a numerical value, adds them up to get a total stream count. The script runs on a 5-second delay loop, keeps track of the peak, and gives me the following output, where the red area is the origin server, green is the repeaters, and blue is the iPhone streams. Windows streams are gleaned directly from Windows Media Services.

Stream count output

Stream count output

That’s all well and good, but how many actual people are watching? We know there are several people who watch this alone, but others do it in groups or with their family. Initially, when we benchmarked other churches, we were told that a ratio of about 1.8 people per stream was a pretty reliable guess. We went with that for a while, as we gathered our own data.

To gather our data, we created a sign-in/feedback form for our web audience that functions very much like the friendship pad we pass around in our physical worship services. One key question that you find only online is “how many people are worshipping today?” Based on the aggregated data from these forms (we’ve collected nearly 8,000), we found that our ratio was closer to 1.7, so we started using that for the purposes of reporting. We typically see about 40% of our peak stream count send us a feedback form, so the per-stream count is probably a fairly representative sample. Periodically, we’ll see the ratio jump up to 2:1 in special circumstances such as inclement weather at our central campus, and we’ll adjust the numbers for that service accordingly.

I’m sure there’s a better way than my cheesy vb script, but it works for now.

Live Streaming on a Budget (Part 5) – Automating EC2

I mentioned a few posts back that I was looking for a way to automate startup and shutdown of the servers. Thanks to some great sleuthing by Justin Moore at Granger Community Church, I got some scripts to start from. I had to make some modifications to suit our exact purposes, but that was relatively easy.

Ingredients:

Linux system with:

  • Java Runtime (in Ubuntu, the package is sun-java6-jre)
  • Amazon EC2 API Tools (installed in /opt/ec2/tools)
  • Wowza Startup Packages (installed in /root/ec2)
  • EC2 keys (installed in /root/ec2)

Note: Because these scripts are run from cron, you’ll need to put all your environment variables to run EC2 at the beginning of each one.

I have 6 separate versions of the startup and termination scripts, one for each server I need to start. I could roll it into one big script, but putting them in their own individual ones not only lets me do an individual machine manually, I can run them all in parallel from cron, which shortens the startup time.

The startup script functions as follows:

  1. Assign environment variables for EC2 and for the machine parameters
  2. Launch machine with ec2-run-instances, redirect output to a temporary file*
  3. Parse temporary file and extract the instance ID, and put it into an environment variable
  4. Write contents of instance ID environment variable to a file in /root/ec2 for use by the shutdown script
  5. Wait 30 seconds
  6. Start checking instance status every so we know when it’s running (wait 10 seconds to check again if it’s not)
  7. Attach any EBS volumes (Optional – I don’t currently need this, so it’s commented out)
  8. Assign Elastic IP
  9. Execute any additional commands via ssh (Optional, I don’t have any that need to run)

* The original scripts use /tmp/a, which is fine, but I had to make each script do its own temporary file since all 6 were running simultaneously and I ran into problems with getting the right Instance IDs set.

The shutdown script works like this:

  1. Query AWS for all running instances
  2. Issue EC2 termination call
  3. ???
  4. PROFIT!

Lastly, put it in your crontab:

# m h  dom mon dow   command
15 8 * * 0 /root/start-windows.sh
25 8 * * 0 /root/start-origin.sh
25 8 * * 0 /root/start-iphone.sh
25 8 * * 0 /root/start-repeater1.sh
25 8 * * 0 /root/start-repeater2.sh
25 8 * * 0 /root/start-repeater3.sh

0 19 * * 0 /root/term-windows.sh
0 19 * * 0 /root/term-iphone.sh
0 19 * * 0 /root/term-origin.sh
0 19 * * 0 /root/term-repeater1.sh
0 19 * * 0 /root/term-repeater2.sh
0 19 * * 0 /root/term-repeater3.sh

This starts up all of them (except the Windows instance which needs more time) at 8:25 on Sunday morning and shuts them down at 7 on Sunday evening.  (be sure that if you’re using GMT on your Linux box to take that into account).

If you’re using an encoder that can be started and stopped on a schedule, synchronize your times with this, and you’ll be golden. The Wowza EC2 images take about 60-90 seconds to fully get up and running, and the Windows one takes about 10-15 min. Currently, the Windows server pulls from the encoder via a 1:1 NAT rule, so the WME instance can be running before the EC2 server is going. When EC2 is ready, it simply connects to the encoder and is off and running.

Live Streaming On a Budget (Part 1) – Genesis

When our senior pastor started casting his vision of an Internet Campus which would revolve around a live (or nearly-live) stream of worship, it became pretty apparent that this was not going to scale well or cheaply. Over the course of the summer of 2008, we started exploring creative ways to do the impossible with nearly no money.

Read More