Mobile Voice in Haiti

November 5, 2013 |   |  Comment  |  Haiti, IT, networking, Wireless

As a follow-on to my previous post about getting mobile internet, here’s one about getting voice service on your US phone (at least if you have a Sprint phone).

I have a Samsung Galaxy S4 on Sprint. Sprint’s CDMA voice network is incompatible with the GSM networks in most of the rest of the world, but recent Samsung Galaxy devices (at least the S3 and S4, and other devices of the same generation/platform) use a software-defined radio that can be made to speak GSM or CDMA at will, with a simple settings change. CDMA doesn’t require a SIM but LTE and GSM do, so the Galaxy is a de facto international phone.

Sprint lets you do international roaming calls for $2/min, which is absurdly high. It’s much better to get a SIM from a local carrier and use that. Making it do this is relatively simple. If your account is in good standing, a simple phone call to Sprint will unlock your phone for using other SIMs (and before you try to do this for a GSM carrier in the US, it explicitly does NOT work on AT&T or T-Mobile). This unlock process does require a data connection (mobile or Wi-Fi) for the phone to receive the unlock signal. After doing that, there’s a simple process that the Sprint rep will give you over the phone to complete the process.

Once that’s done (took me about 5 minutes on the phone – which I did via Skype from Haiti!), all you have to do is go find a local SIM (and in the case of the Galaxy, trim it down to size), pop it in the phone, switch it over to GSM in the Mobile Networks settings, pick your carrier, and off you go.

I’ll add screenshots just as soon as I can make the phone do them. The normal S4 tricks aren’t working.

 

Mobile Internet in Haiti

November 4, 2013 |   |  1 Comment  |  Haiti, Hardware, IT, networking, Wireless

Note: Be sure to read my March 2015 update about this…

I’m back down in Haiti, as some of you already know, working on some of the wireless networks linking the different sites of the Église Méthodiste d’Haïti (EMH), which is the Haitian Methodist Church. Knowing that I was coming into an environment where the internet connection was not functioning properly, and that I was likely going to need internet access for troubleshooting, I armed myself with a 3G GSM hotspot that I picked up on eBay.

After parting with about 50 bucks (plus another 15 for a charger and 2 spare batteries), the Huawei E583C unit showed up via USPS on my doorstep 4 days later bearing a postmark from Hong Kong (color me impressed, I can’t even get postcards from Toronto that quickly!)

20131125_150332I opened it up and inside was a “T-Mobile Wireless Pointer” from the UK division of T-Mobile. I popped on down to the local T-Mobile store and get a SIM for testing, and fired it up. After much futzing around trying to get it to speak 3G to the network without any success, I go back to T-Mobile and pick a tech’s brains. Turns out this one operates on the 800/1800/1900 band, which T-Mobile has phased out 3G on to make room for more LTE. Meanwhile, Jay was in Haiti, so I asked him to pick up a NatCom SIM and bring it home with him.

I’ll pause briefly here to talk a bit about mobile in Haiti. There are two major players, Digicel (which has a thing for island nations all over the world) and NatCom, which is formed out of what was left of the national telephone company (Teleco) and the Vietnamese national telecom (VietTel) that bought up a 70% interest in Teleco not long after the earthquake. What little copper telecom infrastructure existed in the country has long since been destroyed by a number of different Screen Shot 2013-11-25 at 3.20.19 PMmeans, both natural and human. Since the earthquake, NatCom has been building out a LOT of fiber. Digicel operates the only direct fiber link out of the country to Columbus Networks‘ Fibralink fiber network that links the Caribbean up to the rest of the world. The other way out of Haiti to the internet is via microwave backhaul to the Dominican Republic which has 2 landings of the ARCOS fiber ring.

In the nearly 4 years since the quake, mobile internet in Haiti has gone nuts. It’s now quite reliable, and surprisingly cheap if you know how to do it. Monthly postpaid plans for data cost about a quarter what they do in the US – a 10GB plan on digicel will set you back 1000 HTG (about 25 bucks). The same plan on Verizon in the US by comparison is about $100! Digicel offers current-generation Android phones like the S4 (but be prepared to part with full unsubsidized price for it), and Apple recently started making unlocked SIM-less iPhones available on its own store. The smartphone revolution is coming to Haiti, and it’s going to be interesting to watch. There was someone at church on sunday using an iPad, and it wasn’t someone from our team.

When I got down to Haiti and put the SIM Jay obtained for me into the hotspot (erm, “Pointer”… can any Brits enlighten me as to the origin of that term?), and getting no joy. Realizing that the zillion config changes I’d made to try and get it to work on T-Mobile’s network were probably interfering, I hit the factory reset button, and as soon as it rebooted, it was speaking 3G on Natcom’s network. It was that easy.

Next step was to load up some funds on the card, since it was a basic card that came empty of funds. Normally you can do this from the phone, but since this was a hotspot, I didn’t have the ability to dial numbers (although the Huawei firmware does allow you to SMS, which turned out to be a critical component). Natcom partners with a third party called EzeTop which allows you to reload phone cards online (yours or anyone else’s). So I dropped 10 bucks onto it (which translates to 392 HTG, a fairly lousy exchange rate) plus a penny per 10 Goudes as a transaction fee, and off I go. No sign anywhere of what the per-MB cost is. NatCom’s website isn’t particularly helpful in that regard (I later find out that it’s 1.9HTG/MB, about 4 cents.)

Now that I had mobile internet, I fired up the iPad and did some testing on the drive to Petit-Goave, and was getting quite reasonable speeds around 1.5-2Mbps in both directions, very much capable of posting pictures to facebook and whatnot.

Once we got to the guest house where we were staying, we discovered that the wifi there was indeed out of service. I put the hotspot to good use downloading information I was going to need to fix it. In very short order, net access ceases, and I get a screen from NatCom saying that my card is empty, and provides a helpful list of plans and how to activate them. I then go find our hostess and borrow her laptop and internet access to load up some more funds on the card, and then try to activate one of the listed plans. It tells me I can’t do that because I have the wrong type of card.

Then, disaster. Within a matter of little more than an hour, 20 bucks worth of data on the card had vanished. After some digging, I discovered that my good buddy CrashPlan had stabbed me in the back and decided to start a big backup. I killed CrashPlan and reloaded the card (this is getting expensive, and I’m still not entirely sure how much data I’m burning through, especially now that the team is sharing in the internet joy — and the cost!)

Now that I’m back online, I start digging around the NatCom site again to figure out what plans I can access through the SIM I already have. Turns out that they have slightly different SIMs and plans for laptop/USB modems and for mobile phones. I had the latter, a “Nat-Mango” card, which can be had from any street vendor for 25 HTG. I finally found the list of mobile internet plans for the phones, and the correct number to SMS the plan change to. So I send off the text, only to get back “You don’t

Screen Shot 2013-11-05 at 8.03.55 AM

have enough funds for this plan”. I keep moving down the list until even the cheapest one kicks back the message… Uh-oh, I’m running on fumes again. Just as I go to top it up again, it shuts off. Fortunately, one of our Haitian team members had data on his Digicel phone, and I was able to get the account charged up, and switched over to the “Unlimited” plan. Unlimited in this case means 3.5GB at max HSPA+ speeds, then you’re rate-limited to 3.5 Mbps after that. Given that I never saw 3Mbps anywhere, this isn’t really a huge hindrance (that may be a factor of the device more than the network, too). By the time the week was out, our team had gobbled up nearly 25 gigabytes of data through the device.

So, in short, mobile internet from local carriers in Haiti is reliable and cheap (if you know the trick to not paying out the nose per MB), and can be done on a fairly inexpensive piece of hardware. If you’re so inclined, you can also get USB sticks from NatCom for about 1500 HTG. My next step is going to be to see if a device from Cradlepoint can handle the Natcom USB sticks, since they don’t have such a tight limitation on clients.

HLS distribution with Amazon CloudFront

July 5, 2013 |   |  7 Comments  |  Cloud Computing, streaming, Wowza

I’ve blogged extensively about Wowza RTMP distribution with edge/origin and load balancing, but streaming distribution is moving more to HTTP-based systems such as Apple’s HTTP Live Streaming (known inside Wowza as “cupertino”), Adobe’s HTTP Dynamic Streaming (Wowza: “sanjose”), and Microsoft’s Smooth Streaming (Wowza: “smooth”). Future trends suggest a move to MPEG-DASH, which is a standard based on all three proprietary methods (I’ll get into DASH in a future post as the standard coalesces – we’re talking bleeding edge here). The common element in all of them, however, is that they use HTTP as a distribution method, which makes it much easier to leverage CDNs that are geared towards non-live content on HTTP. One of these CDNs is Amazon’s CloudFront service. With edges in 41 locations around the world and 12 cents a gigabyte for transfer (pricing may vary by region), it’s a good way to get into an HTTP CDN without paying a huge amount of money or committing to a big contract with a provider like Akamai.

On the player side, JW Player V6 now supports HLS, and you can do Adobe HDS with the Strobe Media Player.

With the 3.5 release, Wowza Media Server can now act as an HTTP caching origin for any HTTP based CDN, including CloudFront. Doing so is exceedingly simple. First, configure your Wowza server as an HTTP caching origin, and then create a CloudFront distribution (use a “download” type rather than a streaming type – it seems counterintuitive, but trust me on this one!), and then under the origin domain name, put the hostname of your Wowza server. You can leave the rest as defaults, and it will work. It’ll take Amazon a few minutes to provision the distribution, but once it’s ready, you’ll get a URL that looks something like “d1ed7b1ghbj64o.cloudfront.net”. You can verify that the distribution is working by opening a browser to that address, and you should see the Wowza version information. Put that same CloudFront URL in the player URL in place of the Wowza server address, and your players will now start playing from the nearest CloudFront edge cache.

See? Easy.

Browser-Aware Player Code: Episode V, IE Strikes Back

April 17, 2013 |   |  1 Comment  |  Code, Grrr!, streaming, Wowza

Not so long ago, I updated my browser-aware player code to check for the presence of a stream. Recently, it’s come to light that Internet Explorer 9 doesn’t play nice with this particular snippet, because in IE9, the Javascript engine is rather brain-damaged when it comes to cross-site requests. In order to deal with this properly, we must alter the way we query the server for the presence of a stream:


console.log("Starting stream Check");
if (is_ie9) {
console.log ("Using XDR because IE9 is stupid");
streamcheckXDR();
setInterval(function(){streamcheckXDR()},10000);
}
else {
streamcheck();
setInterval(function(){streamcheck()},10000);
}
function streamcheckXDR() {
 console.log("Starting XDR");
 xdr = new XDomainRequest();
 if(xdr) {
 xdr.onerror = function(){ console.log("XDR Error"); };
 xdr.onload = function(){ startPlayer(xdr.responseText,"XDR"); };
 url = "http://"+streamer+":8086/streamcheck?stream="+stream;
 xdr.open("get",url);
 xdr.send();
 } else {
 console.log("Failed to create XDR object");
 }

}

function startPlayer(result,mode){

// for some inexplicable reason, running "Boolean" on the XDR output doesn't work

// so we have to call the function and tell it if we're dealing with XDR data or AJAX data.

 

if(mode == "XDR") {
 if (result === "true") { curstatus = true;}
 if (result === "false") { curstatus = false;}
 } else {
 curstatus = Boolean(result);
 }

//console.log("Result: "+result);
 //console.log("Previous: "+prevstatus);
 //console.log("Current: "+curstatus);
 if (curstatus == prevstatus) {
 //console.log("No Change");
 } else {
 if (curstatus) {
 if (is_iphone || is_ipad || is_ipod) { iOSPlayer("videoframe",plwd,plht,server,stream);}
 else if (is_blackberry) { rtspPlayer("videoframe",plwd,plht,server,stream+'_+240p');}
 else { flashPlayer("videoframe",plwd,plht,server,stream); }
 console.log("Changed from false to true");
 } else {
 var vframe=document.getElementById("videoframe")
 if (is_iphone || is_ipad || is_ipod || is_blackberry) {
 } else {
 jwplayer("videoframe").remove();
 }
 vframe.innerHTML = '<IMG SRC="image.png" WIDTH="'+plwd+'" HEIGHT="'+plht+'">';


 console.log("Changed from true to false");
 }

}
 prevstatus = curstatus;
}

function streamcheck() {
 console.log("Starting AJAX");
 $.ajax({
 dataType: "json",
 contentType: "text/plain",

type: "GET",
 url: "http://"+streamer+":8086/streamcheck?stream="+stream,
 error: function(XMLHttpRequest, textStatus, errorThrown)
 {
 console.log('AJAX Failure:'+ textStatus+':'+errorThrown);
 //some stuff on failure
 },
 success: function(result){startPlayer(result)}
 });
}

Wowza EC2 Capacity Update

February 26, 2013 |   |  4 Comments  |  Cloud Computing, streaming, Wowza

It’s been a while since Wowza has updated their EC2 performance numbers (they date back to about 2009), and both Amazon and Wowza have made great improvements to their products. Since I have access to a high-capacity system outside of Amazon’s cloud, I am able to use Wowza’s load test tool on a variety of instance sizes to see how they perform.

The test methodology was as follows:

  • Start up a Wowza instance on EC2 with no startup packages (us-east)
  • Install the server-side piece of Willow (from Solid Thinking Interactive)
  • Configure a 1Mbps stream in Wirecast
  • Monitor the stream in JWPlayer 5 with the Quality Monitor Plugin
  • Configure the Wowza Load Test Tool on one of my Wowza Hotrods located at Softlayer’s Washington DC datacenter
    • Server is 14 hops/2ms from us-east-1
  • Increase the load until:
    • the measured bandwidth on JW player drops below stream bandwidth
    • frame drops get frequent
    • Bandwidth levels out on the Willow Graphs while connection count increases
  • Let it run in that condition for a few minutes

In Willow, it basically looked like this (this was from the m1.small test). You can see ramping up to 100, 150, 200, 250, 275, and 300 streams. The last 3 look very similar because the server maxed out at 250 Mbps. (Yes, the graph says MBytes, that was a bug in Willow which Graeme fixed as soon as I told him about it)

Willow Bandwidth

Meanwhile, this is what happens on the server.. the CPU has maxed out.

EC2 CPU Usage

So that’s the basic methodology. Here are the results:

[table id=1 /]

There are a couple of things to note here. Naturally, if you’re not expecting a huge audience, stick to m1.small. But the best bang for the buck is the c1.medium (High-CPU Medium), which is a relatively new instance type, which gives you 4x the performance of a m1.small at less than 2x the price. The big surprise here was the m2.xlarge. It performs only marginally better than an m1.small at 4x the price.
All the instances that show 950 are effectively giving you the full benefit of the gigabit connection on that server and maxed out the interface long before the CPU maxes out. In the case of the c1.xlarge, there’s lots of CPU to spare for things like transcoding and such if you’re using a BYOL image. If you want to go faster, you’ll need to roll your own Cluster Quad or do a load-balanced set.

Disclaimers: Your mileage may vary, these are just guidelines, although I think they’re pretty close. I have not tested this anywhere but us-east-1, so if you’re using one of amazon’s other datacenters, you may get different results. I hope to test the other zones soon and see how the results compare.

Wowza/EC2 Q&A Session

February 10, 2013 |   |  3 Comments  |  streaming, Wowza

I’ll admit, the nerd meter on here has been running past redline for a while, so I’m going to step back and answer some questions that have come in via e-mail or twitter from those of you still trying to get started with Wowza streaming.

Q: How do I find a good Wowza EC2 server to use, what should I be using?

A: I understand your confusion. There are a lot of Wowza AMIs out there on the EC2 stack, and that’s just the official ones! When starting an instance from the AWS console, simply searching for “wowza” under the community AMIs is a bit of a daunting process. Just in us-east-1, you get 68 results:

Starting Wowza on EC2

As a general rule, the most recent version available is the one you want to use. Wowza keeps a list of the most recent AMIs on their support site, and unless you have bought a license (Plug: I’m a reseller, support your streamnerd!), you’ll want to use the DevPay instances which require a $5 monthly subscription from Amazon.

If you need to run an older version, Wowza’s support team can give you the AMI ID you need, as they continue to make older versions available (which is why you get such a huge list), although they’re not maintained once a newer AMI has been released. With a few exceptions, startup packages designed for one version will generally work on another version.

Where it gets complicated is if you’re dealing with BYO Licenses and Marketplace instances. That’s probably a topic for another post.

Q: How do I know if I need to set up load balancing and how do I do that, in plain English?

A: Simply put, you’ll need to do load balancing any time the audience on your server exceeds the network throughput that your instance size will support. Wowza has published some numbers in the past for a few of the instance sizes, but those numbers are several years old, and changes to both Amazon’s infrastructure and the Wowza code base have rendered them obsolete. Since I now manage some very large Wowza servers outside the Amazon stack, I will be running some new benchmarking tests in the next few weeks. When I get the results, I’ll be posting those here as well as sharing them with the Wowza team.

The best way to make your Wowza stack scalable is to configure your single server as both an origin and an edge, and make it a load balancing army of one. That way, if you get an unexpected surge in traffic, adding more repeaters is a fairly simple task. All you need to do is keep a set of startup packages with the configuration for each type on hand, and you can start them up.

The key thing to remember about Wowza load balancing is that the load balancer and the origin/edge architecture are independent of each other. In an origin/edge configuration, you publish to one application, and another application (on either the same server or a different server) pull the stream from the origin application and then distribute it out to clients.

There are many ways to determine which edge application a client goes to, but the most common one is the Wowza load balancer module. This module consists of a “sender” that lives on edge servers, and a “listener” that lives on the load balancing server (which is usually the origin server, but doesn’t have to be). The sender keeps track of how many connections are on its server, and reports those back to the listener along with its address and a unique identifier. When queried, the load balancer will then provide the address of whichever server has the fewest connections (or an XML dump listing all the servers in the pool and their status).

Once you have the information of which server has the fewest connections, it’s simply a matter of directing an incoming connection to that edge server. The load balancer package provides an application that will redirect RTMP clients, but that doesn’t work with HTTP clients such as iOS and Android devices, so you’ll need some additional steps for them.

Q: I can check connection counts for stats, but all other posts I read about analytics is confusing. Is there a better, easier way?

One of the most challenging things about Wowza facing new users is that it lacks all the pretty graphs – but the flip side of that is that you get access to raw data. Out of the box, there’s not really anything other than the connectioncounts.xml and serverinfo.xml XML dumps. There are a number of third-party components that can massage this data into something a little more useful and “boss-friendly”. On the commercial side there’s CasterStats (Plug: I’m a reseller. Support your streamnerd!) which can do both live stats and post-analysis of log files, as well as Sawmill and AWStats which will analyze logs. There are probably a few others as well. And, naturaly, there are also a number of PHP scripts that I’ve posted on this blog before that can help with analyzing and processing the XML dumps.

One final plug: If you’re interested in a web-based control system that manages your startup packages and live stats, ask me about cdnBuilder.

That’s all for today, if you’ve got a Wowza question (EC2 or not!), drop me a line on twitter or leave me a comment below!

 

Supercharging a Wowza Hotrod

January 31, 2013 |   |  8 Comments  |  streaming, Wowza

sp1-larry dixon-photo/steve kohlsI’ve posted in the past about running Wowza on large server instances. When running Wowza on systems with a 10Gbps network interface, you run into a 5Gbps limitation inherent to the Java Virtual Machine. With Wowza tuning parameters maxing out at 12 virtual cores, most machines with a 10Gbps interface will have processing power to spare, but why let all that horsepower go to waste?

For relatively little money (under $1000/month), you can get server from 100TB.com with 12 or 16 physical cores (24 or 32 virtual) and park it in the Softlayer datacenter in Washington, DC with a 10Gbps interface, pretty much in the middle of the Internet. With 100TB/month of data transfer included in the price (and for an extra charge, they’ll take the meter off entirely), this is a great option for heavy or full-time streaming. But there’s that pesky JVM throughput limitation. It’s time to take the governor off this hotrod.

What you’ll need:

  • A big honkin’ server
  • A big honkin’ pipe
  • Some extra IP addresses (the 100TB servers at Softlayer come with 8 usable addresses)
  • Wowza with either a subscription license key or two perpetual license keys

Note that I’ve done this in Linux, I’m not quite sure how it would work in Windows with registering the second instance with the services.

Step 1

  1. Install Wowza and tune
  2. Configure it with a license
  3. Choose an IP address on the system
  4. In VHost.xml, and Server.xml replace all instances of <IpAddress>*</IpAddress> with the IP.
  5. Set up load balancing (both sender and listener) with redirect
  6. Add an origin application (type live).
  7. Add an edge application (type liverepeater-edge) pulling from the origin app

Step 2

  • Make a duplicate of the following (I appended a -2):
    • /usr/local/WowzaMediaServer-3.5.0
    • /etc/init.d/WowzaMediaServer *
    • /usr/bin/WowzaMediaServerd *
  • Edit all references to WowzaMediaServer in the files indicated by a *
  • in the cloned installation:
    • change bin/setenv.sh and bin/startup.sh to point to the cloned install
    • delete conf/Server.guid
    • Edit conf/VHost.xml to point to a second IP address
    • Edit conf/Server.xml to point to a second IP address
    • Edit conf/Server.xml to bind jmx to ports 8081/8082
    • Set up load balancing as a Sender (edge) pointing to load balancer IP in primary install
    • Create edge application

Step 3

  • Start the primary instance
  • Check http://primary.ip:1935/loadbalancer?serverInfoXML to verify the primary edge is connected
  • Start the secondary instance
  • Check  load balancer again and make sure secondary instance is connected
  • Publish a stream
  • Pull a stream directly from each edge to make sure the repeaters are working
  • use http://primary.ip:1935/loadbalancer to get least loaded server for load-balancing non-RTMP clients
  • use rtmp://primary.ip/redirect/stream to get load balanced RTMP.

Step 4

  • Sit back, relax, and enjoy the fact that you now have a server that can handle 10,000 connections.
  • If you need more, build up a second server just like this one, and just make both instances edge instances

Browser-aware player code, revisited again

October 29, 2012 |   |  14 Comments  |  Code, streaming, Wowza

It’s the code snippet that just won’t go away. I’ve updated the code for some additional functionality. This version takes server, port, and stream parameters via the URL, parses them in javascript, and then queries a streamcheck HTTPProvider on the server to see if a stream by that name is currently published. If it is, it will load the player, otherwise load a message, and check periodically to see if the stream is published, and load the player if the state changes to true, and unload it if it changes to false, returning to the message. The player is designed to scale to fit whatever window it’s in, so make an IFRAME of whatever size you want the player, and you’re off and running

<IFRAME SRC="player.html?streamer=wowza.nhicdn.net&port=1935&app=live&stream=livestream" WIDTH="640" HEIGHT="360" SCROLLING="NO" />


Without further ado, here’s the code:



<body style="margin: 0px; background: black; color: white; font-family: sans-serif;">
<div id="videoframe" style="text-align: center;font-size: 14;">The video stream is currently offline. Playback will resume as soon as a stream is available.</div>
<script type="text/javascript" src="/assets/jw5/jwplayer.js"></script> 
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>


<SCRIPT type="text/javascript">
// Browser-aware video player for JW Player and Wowza
 

plwd=self.innerWidth;
plht=self.innerHeight;
// var debugwindow=document.getElementById("debug")
// debugwindow.innerHTML='<P>Dimensions: '+plwd+'x'+plht+'</P>';
var streamer=getUrlVars()["streamer"];
var app=getUrlVars()["app"];
var port=getUrlVars()["port"];
var stream=getUrlVars()["stream"];
var server=streamer+':'+port+'/'+app;


var agent=navigator.userAgent.toLowerCase();
var is_iphone = (agent.indexOf('iphone')!=-1);
var is_ipad = (agent.indexOf('ipad')!=-1);
var is_ipod = (agent.indexOf('ipod')!=-1);
var is_playstation = (agent.indexOf('playstation')!=-1);
var is_safari = (agent.indexOf('safari')!=-1);
var is_blackberry = (agent.indexOf('blackberry')!=-1);
var is_android = (agent.indexOf('android')!=-1);
var streamstatus = false;
var prevstatus = false;
var curstatus = false;

streamcheck();
setInterval(function(){streamcheck()},10000);

function streamcheck() {
            $.ajax({
              type: "GET",
              url: "http://"+streamer+":8086/streamcheck?stream="+stream,
             dataType: "json",
             success: function(result){
		curstatus = Boolean(result);
		//if (result === "true") { curstatus = true;}
		//if (result === "false") { curstatus = false;}
		if (curstatus == prevstatus) {
		} else {
		if (curstatus) {
			if (is_iphone || is_ipad || is_ipod) { iOSPlayer("videoframe",plwd,plht,server,stream);}
			else if (is_blackberry) { rtspPlayer("videoframe",plwd,plht,server,stream);}
			else { flashPlayer("videoframe",plwd,plht,server,stream); }
			console.log("Changed from false to true");
		} else {
			var vframe=document.getElementById("videoframe")
			if (is_iphone || is_ipad || is_ipod || is_blackberry) { 
			} else {
				jwplayer("videoframe").remove();
			}
			vframe.innerHTML = 'The video stream is currently offline. Playback will resume as soon as a stream is available.';
			
			
			console.log("Changed from true to false");
		}

		}		
			prevstatus = curstatus;
		}
           });
}

 
function getUrlVars() {
    var vars = {};
    var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi, function(m,key,value) {
        vars[key] = value;
    });
    return vars;
}

function iOSPlayer(container,width,height,server,stream)
{
var player=document.getElementById(container)
player.innerHTML='<VIDEO controls '+
'HEIGHT="'+height+'" '+
'WIDTH="'+width+'" '+
'title="Live Stream">'+
'<SOURCE SRC="http://'+server+'/'+stream+'/playlist.m3u8"> '+
'</video>';
}
 
function rtspPlayer(container,width,height,server,stream)
{
var player=document.getElementById(container)
player.innerHTML='<A HREF="rtsp://'+server+'/'+stream+'">'+
'<IMG SRC="poster-play.png" '+
'ALT="Start Mobile Video" '+
'BORDER="0" '+
'HEIGHT="'+height+'" '+
'WIDTH="'+width+'">'+
'</A>';
}
 
function flashPlayer(container,wide,high,server,stream)
{
jwplayer(container).setup({
		height: high,
		width: wide,
		streamer: 'rtmp://'+server,
		file: stream,
		autostart: true,
		stretching: 'uniform'
		});
	
}
 
</SCRIPT>
</BODY>

The code for the streamcheck module is as follows:

package net.nerdherd.wms.http;

import java.io.*;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

import com.wowza.wms.application.IApplicationInstance;
import com.wowza.wms.http.*;
import com.wowza.wms.logging.*;
import com.wowza.wms.vhost.*;

public class StreamCheck extends HTTProvider2Base {

    	public void onHTTPRequest(IVHost vhost, IHTTPRequest req, IHTTPResponse resp) {
		StringBuffer report = new StringBuffer();
		StringBuffer streamlist = new StringBuffer();
		if (!doHTTPAuthentication(vhost, req, resp))
			return;
		
		Map<String, List<String>> params = req.getParameterMap();
		
		String stream = "";
		boolean status = false;
		boolean listing = false;
				
		if (req.getMethod().equalsIgnoreCase("post")) {
			req.parseBodyForParams(true);
		}
		
		if (params.containsKey("stream"))
			stream = params.get("stream").get(0);
				
		try
		{
				if (vhost != null)
				{
					
					List<String> appNames = vhost.getApplicationNames();
					Iterator<String> appNameIterator = appNames.iterator();
					while (appNameIterator.hasNext())
					{
						try {
	                        String Name = appNameIterator.next();

	                        IApplicationInstance NowApp = vhost.getApplication(Name).getAppInstance("_definst_");
	                        List<String> PublishedNames = NowApp.getPublishStreamNames();
	                        Iterator<String> ThisPublished = PublishedNames.iterator();
	                        if ( PublishedNames.size()>0 )
	                                {
	                                while ( ThisPublished.hasNext() )
	                                        {
	                                        try {
	                                                String NowPublished = ThisPublished.next();
	                                                
	                                                	if (NowPublished.equals(stream)){
	                                                		status = true;
	                                                	}
	                                                	
	                                                 } catch (Exception e) {}
	                                        }
	                                
	                                }
	                        
	                        } catch (Exception e) {report.append(e.toString()); } 
	                        }					
					}
						
								
		}
		catch (Exception e)
		{
			WMSLoggerFactory.getLogger(HTTPServerVersion.class).error("StreamCheck: " + e.toString());
			e.printStackTrace();
		}

		if (!listing) { 
			if (status){
				report.append("true");
		    } else {
		    	report.append("false");
		    }
		}
		
		try {
			resp.setHeader("Content-Type", "text/plain");
			resp.setHeader("Access-Control-Allow-Origin","*");
			OutputStream out = resp.getOutputStream();
			byte[] outBytes = report.toString().getBytes();
			out.write(outBytes);
		} catch (Exception e) {
			WMSLoggerFactory.getLogger(null).error(
					"MediaCasterHTTP: " + e.toString());
		}
		
	
	}
}

Creating a global streaming CDN with Wowza

October 14, 2012 |   |  Comment  |  Cloud Computing, streaming, Wowza

I’ve posted in the past about the various components involved in doing edge/origin setups with Wowza, as well as startup packages and Route 53 DNS magic. In this post, I’ll tie the various pieces together to put together a geographically-distributed CDN using Wowza.

For the purposes of this hypothetical CDN, we’ll do it all within EC2 (although there’s no reason it has to be), with three edge servers in each availability zone.

There are two ways we can do load balancing within each zone. One is to use the Wowza load balancer, and the other is to use round-robin DNS. The latter is not as evenly balanced as the former, but should work reasonably well. If you’re using set-top devices like Roku, you’ll likely want to use DNS as redirects aren’t well supported in the Roku code.

For each zone, create the following DNS entries in Route 53. If you use your registrar’s DNS, you’ll probably want to create a domain name as many registrar DNS servers don’t deal well with third-level domains (sub.domain.tld). For the purposes of this example, I’ll use nhicdn.net, the domain I use for Nerd Herd streaming services. None of these hostnames actually exist.

We’ll start with Wowza’s load balancer module. Since the load balancer is not in any way tied to the origin server (although they frequently run on the same system), you can create a load balancer on one of the edge servers in that zone. Point the loadbalancertargets.txt on each edge server in that zone to that server, and point to the origin stream in the Application.xml configuration file. Once the load balancer is created, create a simple CNAME entry in Route 53 called lb-zone.nhicdn.net, pointing to the hostname of the load balancer server:

lb-us-east.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com

If you opt to use DNS for load balancing, again point the Application.xml to the origin stream, and then create a weighted CNAME entry to each server in that zone, all with the same host record, each with equal weight (1 works):

lb-us-east.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com
lb-us-east.nhicdn.net CNAME ec2-10-20-30-41.compute-1.amazonaws.com
lb-us-east.nhicdn.net CNAME ec2-10-20-30-42.compute-1.amazonaws.com

If you wish, you can also create a simple CNAME alias for each server for easier tracking – this is strictly optional:

edge-us-east1.nhicdn.net CNAME ec2-10-20-30-40.compute-1.amazonaws.com
edge-us-east2.nhicdn.net CNAME ec2-10-20-30-41.compute-1.amazonaws.com
edge-us-east3.nhicdn.net CNAME ec2-10-20-30-42.compute-1.amazonaws.com

Once you’ve done this in each zone where you want servers, Create a set of latency-based CNAME entries for each lb entry:

lb.nhicdn.net CNAME lb-us-east.nhicdn.net
lb.nhicdn.net CNAME lb-us-west.nhicdn.net
lb.nhicdn.net CNAME lb-europe.nhicdn.net
lb.nhicdn.net CNAME lb-brazil.nhicdn.net
lb.nhicdn.net CNAME lb-singapore.nhicdn.net
lb.nhicdn.net CNAME lb-japan.nhicdn.net

Once your DNS entries are in place, point your players to lb.nhicdn.net and use either the name of the edge application (if using DNS) or the redirect application (if using Wowza load balancer), and clients will go to either the edge server or load balancer for the closest region.

One other thing to consider is to set up a load balancer on your origin, and use that for gathering stats from the various servers. You can put multiple load balancers in the loadbalancertargets.txt file, so you can have each server report in to both its regional load balancer and the global one, except the global one is not being used to redirect clients, but rather only for statistics gathering. You can also put multiple load balancers in each region for redundancy and use a weighted CNAME entry.

If you would like assistance setting up a system like this for your organization, I am available for hire for consulting and/or training.

Adding EC2 instances to Route53

September 11, 2012 |   |  10 Comments  |  Cloud Computing

Today’s nifty bit of code is a startup/shutdown script for linux that allows you to add an EC2 instance to Amazon’s Route53 DNS automatically when you start up the instance, and remove it when the instance is knocked down. This script also allows you to add the instance to a weighted round-robin group.

This makes use of the very useful Python-based boto tool which is available in both Yum and Debian repositories under the package name python-boto.

Create this script in /etc/init.d and make it executable, and then add it to the requisite rcX.d directory for startup/shutdown.


#!/bin/bash
#
#       /etc/rc.d/init.d/<servicename>
#
#       Registers DNS with Route 53
#
# chkconfig 345 20 80
#

# Source function library.
. /etc/init.d/functions

# Best practice here is to create an IAM user/group with access to only 
# Route53 and use this keypair.
export AWS_ACCESS_KEY_ID=<Access Key ID Here>
export AWS_SECRET_ACCESS_KEY=<Secret Key ID here>

# Use the domain you have configured in Route 53
export DOMAIN=ianbeyer.com

# This is the hostname of the Weighted Round Robin Group.
export RR=wrr

# Gets the current public hostname - you don't want to use an A record with 
# the IP because the CNAME works in conjunction with Amazon's internal DNS
# to correctly resolve instances to their internal address
export PUB_HOSTNAME=`curl -s --fail http://169.254.169.254/latest/meta-data/public-hostname`

# Gets the Route 53 Zone ID
export ZONEID=`route53 ls | awk '($2 == "ID:"){printf "%s ",$3;getline;printf "%s\n",$3}' | grep $DOMAIN | awk '{print $1}'`

start() {
        echo -n "Registering host with Route 53 : "

# This is the base name of the host you want to use. It will increase the index
# number until it finds one that's not in use, and then use that. 

        export HOST=amz
        export HOSTINDEX=1

        INUSE=`route53 get $ZONEID | grep ${HOST}${HOSTINDEX}\.${DOMAIN} | wc -l`
        while [[ $INUSE > 0 ]]
        do
            HOSTINDEX=$((HOSTINDEX + 1))
            INUSE=`route53 get $ZONEID | grep ${HOST}${HOSTINDEX}\.${DOMAIN} | wc -l`
        done
        if [[ "$HOSTINDEX" == "1" ]]; then
            FQDN="${HOST}${HOSTINDEX}.${DOMAIN}"
        else
            # set the new fqdn hostname and shortname
            FQDN="${HOST}${HOSTINDEX}.${DOMAIN}"
            SHORTNAME="${HOST}${HOSTINDEX}"
        fi

        # Set the instance hostname -- If you want to make sure that bash
        # updates the prompt, run "exec bash" after the script. If you do so
        # in the script, bad stuff happens on startup.

        hostname $FQDN
        echo -n $FQDN


        # Add the DNS record
        RESULT=`route53 add_record $ZONEID $FQDN CNAME $PUB_HOSTNAME | grep "PENDING"`
        if [[ "$RESULT" == "" ]]; then
                echo "... failed.";
        else
                echo "... success.";
        fi

        # Add the CNAME record
        echo -n "Adding host to round-robin group...";

        # Checking to make sure it's not already there. 
        CNAME=`route53 get $ZONEID | grep $RR | grep $PUB_HOSTNAME | wc -l`
        if [[ $CNAME = 0 ]]; then
                RESULT=`route53 add_record $ZONEID $RR.$DOMAIN CNAME $PUB_HOSTNAME 60 ${HOST}${HOSTINDEX} 1 | grep "PENDING"`
                if [[ "$RESULT" == "" ]]; then
                        echo "... failed.";
                else
                        echo "... success.";
                fi
        else
                echo "already exists, ignoring";
        fi

}


stop() {
        echo -n "Deregistering host with Route 53 : "
        HOST=`hostname | cut -f1 -d.`
        FQDN=`hostname`

        # check to make sure it exists in Route53
        CNAME=`route53 get $ZONEID | grep $FQDN | grep $PUB_HOSTNAME | wc -l`
        if [[ $CNAME > 0 ]]; then
                RESULT=`route53 del_record $ZONEID $FQDN CNAME $PUB_HOSTNAME | grep "PENDING"`
                if [[ "$RESULT" == "" ]]; then
                        echo "... failed.";
                else
                        echo "... success.";
                fi
        else
                echo "... not found, ignoring";
        fi

        echo -n "Deregistering host from RR CNAME..."

        # Checking to make sure it exists
        CNAME=`route53 get $ZONEID | grep $RR | grep $PUB_HOSTNAME | wc -l`
        if [[ $CNAME > 0 ]]; then
                RESULT=`route53 del_record $ZONEID $RR.$DOMAIN CNAME $PUB_HOSTNAME 60 ${HOST} 1 | grep "PENDING"`
                if [[ "$RESULT" == "" ]]; then
                        echo "... failed.";
                else
                        echo "... success.";
                fi
        else
                echo "... not found, ignoring";
        fi

        # resets the hostname to default
        hostname `curl -s --fail http://169.254.169.254/latest/meta-data/hostname`


}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo "Usage: <servicename> {start|stop|restart}"
        exit 1
        ;;
esac
exit $?

Many thanks to Dave McCormick for his blog post on the subject from which I borrowed heavily.