Updated the Wowza Launch Script. Changed it to be more friendly to a non-root user directory, as well as adding logic that makes the startup package on the fly, so that if you want to edit the contents, the next launch will send the current incarnation.
Stay tuned for a post soon on the anatomy of a Wowza startup package for EC2.
Updated the code for the Wowza statistics collector. Command-line options, parsing stream names, a little cleanup.
Also updated the RRD pages to reflect the new file I created.
I mentioned in the previous post about using ffmpeg in a cron job to create Simulated Live events via Wowza. In this post, I’ll explain how to do it using the Wowza Stream Class module, which allows you to set a broadcast schedule to play a mix of recorded and live content.
Wowza has a pretty good document on how to add this module in to your server and do a test playlist.If you’re setting this up on Amazon EC2, you’ll need to update your startup package by putting the module in the wowza/lib directory and the playlist in the wowza/content directory
Unfortunately, the tutorial doesn’t really cover playlist creation beyond the example. This is especially tricky, given that the scheduling parameters don’t seem to conform to any known SMIL standard. Yes, it’s XML, so theoretically, it doesn’t matter, but there are extensions in SMIL 3.0 that are meant to deal with server-side playlists for automating programming.
Unless you specified a different application name in the Properties section of Server.xml, the automated playlist will publish to the live application.
The basic structure of the SMIL file body consists of <stream> and <playlist> statements.
The <stream> element defines one or more virtual stream names that the playlists will feed into:
<stream name="playlist-high"></stream> <stream name="playlist-low"></stream>
In this example, I have a high and low bandwidth stream. In your player, you reference the stream name, rather than the streamshedule.smil file, like this:
streamer: rtmp://wowza.server.address(:port)/live
file: playlist-high
http://wowza.server.address/live/playlist-high/manifest.f4m
http://wowza.server.address/live/playlist-high/playlist.m3u8
http://wowza.server.address/live/playlist-high/Manifest
The <playlist> element defines specific video sequences that go into the virtual stream. There are four key parameters to the playlist element:
Within the playlist element are one or more <video> statements that use the following parameters:
Using start/length is a useful way to introduce commercial breaks or intermissions into a stream. This example would show BigBuckBunny.mp4 from the start, for 60 seconds, then cut to a commercial for the duration of the advertisement-1.mp4 file. After the commercial, it would resume and play for 2 more minutes, play a 30-second commercial from advertisement-2.mp4 and then plays the rest of the BigBuckBunny.mp4 file. If the playlist set to repeat, this will loop.
<video src="mp4:BigBuckBunny.m4v" start="0" length="60"/> <video src="mp4:advertisement-1.mp4 start="0" length="-1"/> <video src="mp4:BigBuckBunny.m4v" start="60" length="120"/> <video src="mp4:advertisement-2.mp4 start="0" length="30"/> <video src="mp4:BigBuckBunny.m4v" start="180" length="-1"/>
When a particular playlist has ended, and there are no others currently scheduled, it will default to the last playlist, even if that playlist’s repeat is set to false.
Here’s an example for our weekly service replays, and live sunday events:
<smil> <head> </head> <body> <stream name="playlist-high"></stream> <stream name="playlist-low"></stream> <playlist name="mon-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-17 07:45:00"> <video src="mp4:traditions-l.mp4" start="0" length="-1"/> </playlist> <playlist name="mon-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-17 07:45:00"> <video src="mp4:traditions-h.mp4" start="0" length="-1"/> </playlist> <playlist name="mon-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-17 09:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="mon-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-17 09:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="tue-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-18 12:45:00"> <video src="mp4:praise-l.mp4" start="0" length="-1"/> </playlist> <playlist name="tue-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-18 12:45:00"> <video src="mp4:praise-h.mp4" start="0" length="-1"/> </playlist> <playlist name="tue-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-18 14:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="tue-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-18 14:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="wed-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-19 21:45:00"> <video src="mp4:praise-l.mp4" start="0" length="-1"/> </playlist> <playlist name="wed-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-19 21:45:00"> <video src="mp4:praise-h.mp4" start="0" length="-1"/> </playlist> <playlist name="wed-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-19 23:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="wed-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-19 23:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="thu-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-20 03:15:00"> <video src="mp4:traditions-l.mp4" start="0" length="-1"/> </playlist> <playlist name="thu-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-20 03:15:00"> <video src="mp4:traditions-h.mp4" start="0" length="-1"/> </playlist> <playlist name="thu-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-20 05:00:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="thu-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-20 05:00:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="fri-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-21 07:45:00"> <video src="mp4:traditions-l.mp4" start="0" length="-1"/> </playlist> <playlist name="fri-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-21 07:45:00"> <video src="mp4:traditions-h.mp4" start="0" length="-1"/> </playlist> <playlist name="fri-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-21 09:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="fri-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-21 09:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="sat-l" playOnStream="playlist-low" repeat="false" scheduled="2011-01-22 02:45:00"> <video src="mp4:praise-l.mp4" start="0" length="-1"/> </playlist> <playlist name="sat-h" playOnStream="playlist-high" repeat="false" scheduled="2011-01-22 02:45:00"> <video src="mp4:praise-h.mp4" start="0" length="-1"/> </playlist> <playlist name="sat-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-22 04:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="sat-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-22 04:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="sun-am-high" playOnStream="playlist-high" repeat="false" scheduled="2011-01-23 11:30:00"> <video src="mobile-2" start="-2" length="6300"/> </playlist> <playlist name="sun-am-low" playOnStream="playlist-low" repeat="false" scheduled="2011-01-23 11:30:00"> <video src="mobile-1" start="-2" length="6300"/> </playlist> <playlist name="sun-am-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-23 13:15:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="sun-am-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-23 13:15:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="sun-pm-high" playOnStream="playlist-high" repeat="false" scheduled="2011-01-23 17:45:00"> <video src="mobile-2" start="-2" length="6300"/> </playlist> <playlist name="sun-pm-low" playOnStream="playlist-low" repeat="false" scheduled="2011-01-23 17:45:00"> <video src="mobile-1" start="-2" length="6300"/> </playlist> <playlist name="sun-pm-loop-h" playOnStream="playlist-high" repeat="true" scheduled="2011-01-23 19:30:00"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="sun-pm-loop-l" playOnStream="playlist-low" repeat="true" scheduled="2011-01-23 19:30:00"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> <playlist name="default-high" playOnStream="playlist-high" repeat="true" scheduled="2011-01-01 00:00:01"> <video src="mp4:1-16-Loop-H.mp4" start="0" length="-1"/> </playlist> <playlist name="default-low" playOnStream="playlist-low" repeat="true" scheduled="2011-01-01 00:00:01"> <video src="mp4:1-16-Loop-L.mp4" start="0" length="-1"/> </playlist> </body> </smil>
Video files can either be uploaded, or recorded on the server using a live-record application type.
Once your playlist is built, you’ll need to restart the Wowza service for it to read the new playlist in and schedule it internally.
Update(July 18, 2001) : I’ve added a post about my Excel playlist generator.
Last summer, we switched our primary Flash streaming over to 316 Networks partly because of the simulated live capability they offered, and partly for the Media Suite backend. We continued to use Wowza on EC2 for our mobile users, since the solution works very well. Unfortunately, simulated doesn’t work for our mobile users, who are limited to live.
We have 3 replays a week for each of our two web services: the traditional service from Sunday morning, and the praise service from Sunday night.
I should probably step back for a moment and explain what “Simulated Live” means. It’s a recorded event, but from a user standpoint, it behaves like a live event. There’s a set start time, and if you come in 20 minutes after the start of the event, you get the video 20 minutes in; there are no “trick play” DVR functions like fast-forward or rewind. This gives a shared experience for everyone who is watching, and also keeps us legit with the copyright restrictions, as our “live” events are considered extensions of the actual live event in the sanctuary. Simulated live is also known as “pseudostreaming”
Wowza doesn’t natively support pseudostreaming (although the Stream Class API does have some scheduling capability — see this post), so we needed some way of broadcasting recordings on a schedule. I could use the VT5 machine and the Scheduler to replay the recordings on a schedule, but the big downside is that it consumes local bandwidth, which is in short supply.
What I needed was a way of streaming the archive files (high and low bandwidth) that were created by the Kulabyte encoder during the live event. Ideally, since I already upload these to 316 for rebroadcast, I’d like to not have to upload it to two locations.
Enter the lovely open-source encoder, ffmpeg. My concern about using ffmpeg was that re-encoding an already encoded file had the potential of introducing compression artifacts and adding CPU load. I was very happy to find an obscure command-line setting that tells it to copy the input file’s audio and video codec directly. The only thing ffmpeg would be doing would be extracting the audio and video streams from the MPEG-4 container and stuffing them into a Flash container without molesting the actual streams at all. Added bonus: ffmpeg can also not only output to RTMP, but can take RTMP as an input as well.
Fortunately, Media Suite’s media bin makes videos available via direct RTMP and HTTP and helpfully provides the CDN URL for those files. Another method of storage would be to use Amazon S3 and get the files either via HTTP or Cloudfront RTMP.
Attempt #1: Installed ffmpeg on a local Ubuntu box via apt-get, no dice. It refused to connect to the RTMP server. After some research, I found that the version of ffmpeg on the Jaunty version of Ubuntu is 0.50, and seems to have some weird build options.
Attempt #2: I downloaded the ffmpeg 0.61 source, ran a quick configure/make/make install on it, and tried again. Success! I was taking an MP4 recording on a disk, and streaming to Wowza. I then changed the input from a local file to to the URL provided by 316, and weird stuff started happening. And then I realized that the URL had some GET parameters in it that left a question mark and two ampersands in the URL that I needed to escape before bash would parse it correctly. Once I fixed that, it started running happily, and I was seeing the stream on my iPod.
Attempt #3: I Downloaded the ffmpeg 0.61 source to my Wowza server on EC2 and crossed my fingers that the build would go off without a hitch. Success! I then transplanted the command line I used on my test box, tweaked the destination server to the local Wowza install, and fired it off. Success again!
Now that I’d proved the concept, it was simply a matter of putting that command into a cron job and waiting to see if it fired off. And then realizing that the server is in eastern time. And then realizing that I should probably correctly specify the path to ffmpeg (doh!). But once I got those silly details ironed out, off it went.
So now I have ffmpeg on my EC2 system, consuming virtually no CPU, pulling my archive files from 316 (only had to upload them once), streaming to Wowza on a scheduled basis, without chewing up T1 bandwidth at our main site.
Unfortunately, there’s a little bit of brain damage involved in pulling it from 316, because I’ll have to go change the filename in the cron job every week. Perhaps I’ll end up uploading it to S3 after all and just giving it a static name.
Here’s the commandline used:
/path_to/ffmpeg -i rtmp://rtmp.server/filename.mp4 -re -sameq -acodec copy -vcodec copy -f flv rtmp://localhost/live/stream
Commandline options used:
(or, How Amazon Cloudwatch helps manage Wowza server load)
This morning I woke up to two things: Beautiful Kansas City February weather (aka, an ice storm), and a voicemail from the Senior Pastor, asking if we had sufficient online capacity to support a larger-than-usual stream audience. Online worship streaming is a great option for these weather events that have been so common this winter (and not just in the KC area – we see increased online attendance when the weather gets foul elsewhere, like the DC storms of a few weeks ago).
My first indication that this was going to be a big event was Woopra showing 30 people on the web page half an hour before we start sending any kind of video (which is itself 75 minutes before we actually start the morning service). Usually there are two or three. Fifteen minutes after we started sending video, we were already cranking out 20-30 streams (again, we usually only have a small handful at this point).
AWS CPU Usage
Most weeks, we run two Wowza repeaters pulling from a single origin server, which gives us plenty of capacity. I had to spin up a third repeater by the beginning of the pre-service music, a fourth about 10 minutes later, and a fifth after five more minutes. I set my threshold for spinning a new server at 75% CPU on the repeaters, as indicated by the AWS CloudWatch monitors. In the case of a heavy influx of viewers, this gives the new instance enough time to get up and running before the other repeaters hit 100% CPU. Wowza tells me this is at about 180Mbit/sec on a small instance, which for us means around 300 streams. The CPU threshold of 75% works out to about 260 streams.
Unfortunately for our online worshipers, our web server was bogging down pretty hard at
Web Server CPU usage
the beginning of the service, where the two CPU cores were maxed out for about 15-20 minutes, which translated into slower page loads. The database server wasn’t sweating too hard, so I suspect this could have been helped with better PHP caching. Fortunately for me, this had the effect of slowing down the rate of incoming streams, which allowed me to get new repeaters going before the existing ones started choking.
You can see in the graph where we added new repeaters, and how fast they ramped up. It also shows how incredibly well Wowza’s built-in load balancing works. We eventually leveled out at a little over 1100 streams, which meant our EC2 instances were cranking out 600-700 Mbps for nearly an hour:
AWS Network Usage
Meanwhile, this is what we were seeing on Woopra (note the fortunate souls escaping the ice storm in Aruba and the Cayman Islands!):
Next step is to define rules in Cloudwatch for automatically scaling. For that to work, I’m going to need to build my own Wowza AMI, since the current method of starting repeaters involves sending the instance a startup package from the client. I’ll need to build this configuration into the server for CloudWatch scaling to work properly.
Longtail recently released JW Player 5.0, but it had a bug that prevented it from being used with a Wowza load balance setup. It would catch the redirect and show the first few frames and then start buffering without end.
I just got the new 764 build and am happy to report that it works quite nicely now.
Since there’s been a flurry of interest lately on my series of posts on live church streaming, here’s an index to the entire set of posts:
For the benefit of the less technically inclined, I’ve moved scripts and code to their own section and updated posts that contained the code with links to the appropriate page.
Back in July, I made a post about metrics and a cheesy VB Script that got the job done, but wasn’t particularly elegant. I’ve since improved on this due to load balancing (I posted about that in September).
I’ve since then learned a bunch about RRDTool, and have put together a script that pulls the XML data, groks it, and then populates an RRD. The net result is that I get a graph like this:
This graph gives me the following information: The iPhone stream count (with a 10:1 vertical exaggeration), the Flash stream count, and the total viewer count, which is the sum of Flash and iPhone streams, multiplied by a factor of 1.7 (which we’ve found reasonably reflects how many actual people are watching, versus streams. Then the vertical red line shows the time the peak occurred, and the horizontal line shows the level of that peak. The actual peak numbers are listed on the bottom. The RRD and the graph are set up to take into account Windows streams, just as soon as I find a good way to pull that data from WMI via perl.
The general operation is as follows: There’s a script that’s started in a cron job 10 minutes after the servers are spun up, and it polls the origin server every 10 seconds for its counts and populates the RRD. There’s another cron job that runs every minute to generate a current graph, which is then displayed on a page with some javascript that refreshes the image in realtime. Then, at 12:15 and 6:30, there’s another cron job that takes a snapshot of the previous two hours, puts it into a web-accessible directory, and appends an HTML file with a link for easy access later.
All of the metrics scripts, automation scripts, and graphing tools live on a linux virtual machine that runs on our central campus.
This is big improvement over dumping a vbscript into a CSV and then graphing manually with Excel. This happens automatically, in real time, without me being there.