EC2 Monitoring with Raspberry Pi

May 4, 2017 |   |  2 Comments  |  Cloud Computing, Code, Hardware, nerd, networking

I’ve been doing a little Raspberry Pi hacking lately, and put together a neat way to have physical status LEDs on your desk for things like EC2 instances.

The Hardware

In its most basic form, you can simply hook up an LED and a bias resistor between a ground line and a GPIO line on the Pi, but that doesn’t scale especially well – You can run out of GPIO lines pretty quickly, especially if you’re doing different colors for each status. Plus, it’s not overly elegant.

The solution? Unicorns!

No, really. The fine folks at Pimoroni in Sheffield, UK have made a lovely little HAT device for the Pi called a Unicorn. Its primary purpose is lots of blinky lights to make pretty rainbows and stuff, hence the name. However, this HAT is a 4×8 (or an 8×8) array of RGB LEDs, addressable via the I2C bus, which doesn’t eat up a line per LED (good thing, otherwise it would require 96 analog lines). The unicornhat library (python3-unicornhat) is available for Python 2 and Python 3 in the Raspbian repo. When installed onto the Pi, the Unicorn will fit within a standard Raspberry Pi case.

The Code

This is my first foray into Python, so there was a bit of a learning curve. If you’re familiar with object-oriented code concepts, this should be easy for you. Python is much more parsimonious with punctuation than PHP or perl are.

For accessing the EC2 data, we’ll need Amazon’s boto3 library, also available in the Raspbian repo (python3-boto). One area where boto3 is really nice is that the data is returned directly as a dict object (what users of other languages would call an array), so you don’t have to mess with converting JSON or XML into an object structure, and it can be manipulated as you would any other associative array (or a hash for you old-timers that use perl). AWS returns a fairly complex object, so you kind of have to dig into it via a few iterative loops to extract the data you’re after.

From there, it’s a matter of assigning different RGB values to the states. I chose these ones:

  • stopped: red
  • pending: green
  • running: blue
  • stopping: yellow(ish)

I also discovered that I needed to assign a specific pixel to each instance ID, otherwise they tended to move around a bit depending on what order AWS returned them on a particular request.

Here’s what the second iteration looks like in action:

import boto3 as aws
import unicornhat as unicorn
import time

# Initialize the Unicorn
unicorn.clear()
unicorn.show()
unicorn.brightness(0.5)

# Create an EC2 object 
ec2 = aws.client('ec2')

# Define colors and positions
color = {}
color['stopped']={'red':255,'green':0,'blue':0}
color['pending']={'red':64,'green':255,'blue':0}
color['running']={'red':32,'green':32,'blue':255}
color['stopping']={'red':192,'green':128,'blue':32}
	
pixel = {}
pixel['i-0fa4ea2560aa17ffd']={'x':0,'y':0}
pixel['i-06b95cd864acb1a8c']={'x':0,'y':1}
pixel['i-0661da0f50ffb604c']={'x':0,'y':2}
pixel['i-063ec151e0f44ef9b']={'x':0,'y':3}
pixel['i-02c514ca567d8a033']={'x':0,'y':4}

# Loop until forever
while True:

	response = ec2.describe_instances()
		
	
	statetable = {}
	resarray = response['Reservations']
	for res in resarray:
		instarray = res['Instances']
		for inst in instarray:
			iid = inst['InstanceId']
			state = inst['State']['Name']
			# print(iid)
			# print(state)
			statetable[iid] = state
	
	
	for ec2inst in statetable:
		x = pixel[ec2inst]['x']
		y = pixel[ec2inst]['y']
		r = color[statetable[ec2inst]]['red']
		g = color[statetable[ec2inst]]['green']
		b = color[statetable[ec2inst]]['blue']
		# print(x,y,r,g,b)
	
		unicorn.set_pixel(x,y,r,g,b)
		unicorn.show()


	time.sleep(1)

For the moment, this is just monitoring EC2 status, but I’m going to be adding checks in the near future to do things like ping tests, HTTP checks, etc. Stay tuned.

Multi-tenant Virtual Hosting with Wowza on EC2

May 22, 2016 |   |  Comment  |  Cloud Computing, networking, streaming

That’s a mouthful, isn’t it?

I recently needed to migrate a couple of Wowza Streaming Engine tenants on a baremetal server that was getting long in the tooth, and was getting rather expensive. These tenants were low-volume DVR or HTTP transmuxing customers, with one transcoding customer that required some more CPU power. But this box was idle most of the time. So I decided to move it over to AWS and fire up the box only when necessary. Doing this used to be a cumbersome process with the AWS command-line tools that were Java-based. The current incarnation of tools is quite intuitive and runs in Python, so there’s not a lot of insane configuration and scripting to do.

You may recall my post from a few years back about multi-tenant virtual hosting. I’m going to expand on this and describe how to do it within the Amazon EC2 environment, which has historically limited you to  a single IP address on a system.

The first step to getting multiple network interfaces on EC2 is to create a Virtual Private Cloud (VPC) and start your EC2 instances within your VPC. “Classic” EC2 does not support multiple network interfaces.

Once you’ve started your Wowza instance within your VPC (for purposes of transcoding a single stream, I’m using a c4.2xlarge instance), you then go to the EC2 console, and on the left-hand toolbar, under “network and security” is a link labeled “Network Interfaces”. When you click on that, you have a page listing all your active interfaces.

To add an interface to an instance, simply create a network interface, select the VPC subnet it’s on, and optionally set its IP (the VPC subnet is all yours, in dedicated RFC1918 space, so you can select your IP). Once it’s created, you can then assign that interface to any running instance. It shows up immediately within the instance without needing to reboot.

Since this interface is within the VPC, it doesn’t get an external IP address by default, so you’ll want to assign an ElasticIP to it if you wish to have it available externally (in most cases, that’s the whole point of this exercise)

Once you have the new interface assigned, simply configure the VHosts.xml and associated VHost.xml files to listen to those specific internal IP addresses, and you’re in business.
As for scheduling the instance? On another machine that IS running 24/7 (if you want to stick to the AWS universe, you can do this in a free tier micro instance), set up the AWS command line tools and then make a crontab entry like this:

30 12 * * 1-5 aws ec2 start-instances --instance-ids i-XXXXXXXX
35 12 * * 1-5 aws ec2 associate-address --network-interface-id eni-XXXXXXXX --allocation-id eipalloc-XXXXXXXX
35 12 * * 1-5 aws ec2 associate-address --network-interface-id eni-XXXXXXXX --allocation-id eipalloc-XXXXXXXX
30 15 * * 1-5 aws ec2 stop-instances --instance-ids i-XXXXXXXX

This fires up the instance at 12:30pm on weekdays, assigns the elastic IPs to the interfaces, and then shuts it all down 3 hours later (because this is an EBS-backed instance in a VPC, stopping the instance doesn’t nuke it like terminating does, so any configuration you make on the system is persistent)

Another way you can use this is to put multiple interfaces on an instance with high networking performance and gain the additional bandwidth of the multiple interfaces (due to Java limitations, there’s no point in going past 4 interfaces in this use case), and then put the IP addresses in either a round-robin DNS or a load balancer, and simply have Wowza bind to all IPs (which it does by default).

Inside Wowza Startup Packages

January 26, 2011 |   |  6 Comments  |  Internet Campus, streaming, Wowza

Startup packages are one of the more useful features of Wowza Media Server for EC2 – they allow you to custom-configure a system for rapid scaling and provisioning. Wowza provides several starter packages to build on.

A startup package is a file (up to 16384 bytes in size) that’s passed to the instance through the –user-data-file parameter on the API tools (if you’re uploading it via the AWS Web Console, you’ll need to encode it to Base64 and paste it into the text box) . There are a few ways that the data can get into the instance, which Amazon documents over here. For a generic EC2 instance, this can be anything, from text to binary data, depending on what the processing on the target instance is set up to do. In the case of Wowza, it’s a zip file with a specific structure. Much of this is digested from the Wowza for EC2 guide.

File Contents

A startup package for EC2 contains the following:

  • startup.xml (startup manifest)
  • tuning folder
  • wowza folder
  • Any other folders referenced in the startup manifest

A startup package is limited to a maximum of 16KB.

Startup activities are logged to /usr/local/WowzaMediaServer/logs/wowzamediaserver_startup.log. This is a good place to look if it’s not behaving as expected. The package is unpacked to /opt/working.

Startup Manifest

This file controls the startup processing for instantiating a Wowza server on Amazon EC2. It allows three commands: Install, Download, and RunScript.

Download

The <Download> command will download content from a web server and save it to the local Amazon instance. The <Download> command includes the following elements: URL, Data, Header, Destination, and Action:

<Download>
<URL>[URL]</URL>
<Data>[data]</Data>
<Header><Name>[key-name]</Name><Value>[value]</Value></Header>
<Header><Name>[key-name]</Name><Value>[value]</Value></Header>
<Destination>[relative-or-absolute-file-path]</Destination>
<Action>[UNZIP, INSTALL]</Action>
</Download>

The only two required elements are <URL> and <Destination>. To download a file from the url http://www.mycompany.com/myfile.zip, save it to the local machine at the location /opt/myfile.zip and unzip the file after download, the command is:

<Download>
<URL>http://www.mycompany.com/myfile.zip</URL>
<Destination>/opt/myfile.zip</Destination>
<Action>UNZIP</ Action >
</Download>

When completed, the contents of the zip archive are located in /opt.

One use of the <Download> command is to work around the 16kB startup package size limitation. For example, if you need to add several .jar files into the Wowza Server “lib” folder and these files push your startup package size over the 16kB limit, you might package these files into a separate zip archive. You can then host this zip archive on a web server and use the <Download> command to install the files into the Wowza Server “lib” folder.

It’s important to remember that the zipfile path structure is critical. If it uses no paths, you’ll need your destination to be where it ultimately lives, either in the staging area, or the absolute path to the Wowza install. When creating the zipfile with relative paths, create the path tree as if you were in the Wowza installation root.

URL

The <URL> is the URL of the file to be downloaded. The download can be performed over SSL by starting the url with https:// rather than http://. The url can also contain query parameters. The file will be downloaded using the GET method unless <Data> is specified.

Data

The Data is text data that will be included as part of the body of the HTTP request. You can use post data to send user name and password information to your web server so you can protect your content.

Header: <Name> and <Value>

The <Header> elements are name value pairs added to the header part of the HTTP request. An example would be:

<Header>
<Name>Content-type</Name>
<Value>text/plain</Value>
</Header>

Destination

The <Destination>element is the path to which the file will be saved (including the filename). This path can be relative or absolute. The base directory when calculating a relative file path, is the root directory of the startup package (the folder that contains the startup.xml file).

Action

The <Action> element is the action performed after the file is downloaded. The action can either be UNZIP or INSTALL. If the action is UNZIP the downloaded file will be unzipped using the unzip command. If the action is INSTALL the downloaded file will be unzipped and the contents of the folder will be installed (copied) into the Wowza Server installation folder.

Install

The <Install> command will copy the contents of a folder into the Wowza Server installation folder. The <Install> command can either contain a single <Package> element or single <Folder> element.

<Install>
<Package>[path-to-package]</Package>
</Install>
<Install>
<Folder>[foldername]</Folder>
</Install>

The Package path can reference an external URL, like http://wowzamediasystems.s3.amazonaws.com

The Folder path can reference either a relative path (relative to the root of the startup package where Startup.xml is located) or an absolute path on the local file system.

RunScript

The <RunScript> command will execute a script on a running Amazon instance.

<RunScript>
<Script>[relative-or-absolute-file-path]</Script>
<Param>[parameter]</Param>
<Param>[parameter]</Param>
</RunScript>

Script

The <Script> element is the path to the script file to be executed. This path can be relative or absolute. The base directory when calculating a relative file path, is the root directory of the startup package (the folder that contains the startup.xml file). This can refer to a script, or be a single-line command.

Any files referenced in the script need absolute paths or a path relative to the startup package root.

Param

The <Param> elements are parameters that will be passed to the running script. For example the following <RunScript> command:

<RunScript>
<Script>scripts/copyfile.sh</Script>
<Param>filea.txt</Param>
<Param>fileb.txt</Param>
</RunScript>

Would be the equivalent of executing the command:

./scripts/copyfile.sh filea.txt fileb.txt

Environment Variables

The following environment variables are available to scripts launched by the startup processor:

AWSEC2_METADATA_INSTANCE_ID - Amazon instance id
AWSEC2_METADATA_SECURITY_GROUPS - Security group
AWSEC2_METADATA_LOCAL_IPV4 - Local IP address
AWSEC2_METADATA_AMI_LAUNCH_INDEX - Launch index
AWSEC2_METADATA_PUBLIC_HOSTNAME - Public host name
AWSEC2_METADATA_PRODUCT_CODES - DevPay product code
AWSEC2_METADATA_INSTANCE_TYPE - instance type (m1-small, m1-large, m1-xlarge)
AWSEC2_METADATA_HOSTNAME - Public host name
AWSEC2_METADATA_LOCAL_HOSTNAME - Local host name
AWSEC2_METADATA_PUBLIC_IPV4 - Public IP address
AWSEC2_METADATA_AMI_MANIFEST_PATH - S3 manifest path
AWSEC2_METADATA_RESERVATION_ID - Instance reservation ID
AWSEC2_METADATA_AMI_ID - AMI ID

Wowza Folder

The Wowza folder in the startup package is meant to mirror the Wowza installation folder on the server. When the Install command in the startup manifest is invoked with this folder, the file structure of this folder will be copied to the installation folder.

Applications Tree

Contains folders for each Wowza application configured. There are not typically any files in this tree, just folders.

Conf Tree

Contains the configuration files for the server (at the root of the tree) and for each application (in folders matching the applications tree

Content Tree

Contains any content referenced by the applications. This is where SMIL files, stream schedules, and such go. Any audio/video content that goes here won’t fit in the startup package and will need to be downloaded separately.

Lib Tree

Contains any additional modules for the server.

Tuning Folder

This folder contains tuning scripts that tune the Wowza server.  It copies the requisite environment variables and tuning commands to a script executed as part of the Wowza startup, which happens after the startup processor is run.

Scripting

The following useful linux tools are available on the standard EC2 build (based on Fedora):

  • MySQL client commands
  • zip/unzip, bzip/bunzip, gzip/gunzip
  • s3fs
  • Compiler and build tools
  • WGet
  • RSync
  • SSH commands
  • dos2unix/unix2dos
  • curl
  • perl 5.8.8 (with MySQL support)
  • GPG
  • Shells: bash, sh
  • RRDTool
  • PHP5
  • EC2 API tools

Services

The following services are available on the EC2 build of Wowza, in startup order:

  • SNMP
  • SSH
  • FTP (Anonymous access to /var/ftp/)
  • MySQL (default password = “password”)
  • Wowza
  • Apache 2 – port 8080, content in /var/www/html
    • Cacti
  • Java Console

Startup package scripts and data are invoked by the Wowza startup script. If you modify any applications started prior to that, you’ll need to restart them.

Example

Here’s what my startup.xml looks like:

<Startup>
 <Commands>

 <!--
  Comments
  -->

 <Download>
  <URL>http://webserver/wowza/wms-plugin-collection.zip</URL>
  <Destination>wowza/lib/wms-plugin-collection.zip</Destination>
  <Action>UNZIP</Action>
 </Download>

 <RunScript>
  <Script>scripts/mount-s3.sh</Script>
 </RunScript>

 <Install>
  <Folder>wowza</Folder>
 </Install>

 <RunScript>
  <Script>tuning/tune.sh</Script>
 </RunScript>

 <RunScript>
 <Script>scripts/enable_cacti.sh</Script>
 </RunScript>
 </Commands>
</Startup>

How this works:

  • Download the wms-plugin-collection.zip file and dump it in the staging area for wowza (/opt/working/wowza/lib)
  • Unzip it (this leaves the zip file behind, but that doesn’t matter)
  • Run the a script that mounts some s3 buckets and copies them into the content folder:
#!/bin/sh
mkdir -p /usr/local/WowzaMediaServer/content/s3
mkdir -p /usr/local/WowzaMediaServer/content/archives

s3fs bucket1 -o accessKeyId=XXX -o secretAccessKey=YYY /usr/local/WowzaMediaServer/content/s3

s3fs bucket2 -o accessKeyId=XXX -o secretAccessKey=YYY /usr/local/WowzaMediaServer/content/archives/

cp /usr/local/WowzaMediaServer/content/s3/* /usr/local/WowzaMediaServer/content
  • Install the Wowza configs from the staging area
  • Run the tuning scripts
  • Run a script that automatically enables Cacti (since polling the local host is disabled by default):
#!/bin/sh
mysql -u root -ppassword < scripts/enable_cacti.sql

enable_cacti.sql contains the following statement:

update cacti.host set disabled='' where id='2';

(note that if you’re using an Elastic IP, you’ll need to restart the Wowza Service for Cacti to behave)

In my Wowza directories, I have:

  • applicationsdirectory
    • live directory (think of this as a mount point – it’s an empty directory, but has to exist)
  • confdirectory
    • Server.xml (general server parameters)
    • VHost.xml (host bindings, HTTP providers, etc.)
    • livedirectory (this is the application configuration for the live application)
      • Application.xml (defining the live application)
  • content directory
    • ipad.smil ( multi-bitrate stream selection for iOS devices)
    • mobile.smil (defining where the Roku stream goes – this abstracts a potentially changing stream name, as well as giving me a way to track Roku traffic using this perl stats collection script)
    • streamschedule.smil (defines the schedule for the Stream Class module)
    • Additional material is pulled from S3 in the aforementioned script
  • lib directory (empty mount point)

To start my Wowza instances, I create the startup package file and tree structure, and then call this startup script that packs up the zip file and fires off the instance.

Anatomy of an online worship service

February 21, 2010 |   |  4 Comments  |  Internet Campus, streaming, Wowza

(or, How Amazon Cloudwatch helps manage Wowza server load)02-21-10-AM-AWS

This morning I woke up to two things: Beautiful Kansas City February weather (aka, an ice storm), and a voicemail from the Senior Pastor, asking if we had sufficient online capacity to support a larger-than-usual stream audience. Online worship streaming is a great option for these weather events that have been so common this winter (and not just in the KC area – we see increased online attendance when the weather gets foul elsewhere, like the DC storms of a few weeks ago).

My first indication that this was going to be a big event was Woopra showing 30 people on the web page half an hour before we start sending any kind of video (which is itself 75 minutes before we actually start the morning service). Usually there are two or three. Fifteen minutes after we started sending video, we were already cranking out 20-30 streams (again, we usually only have a small handful at this point).

02-21-10-AM-CPU

AWS CPU Usage

Most weeks, we run two Wowza repeaters pulling from a single origin server, which gives us plenty of capacity. I had to spin up a third repeater by the beginning of the pre-service music, a fourth about 10 minutes later, and a fifth after five more minutes. I set my threshold for spinning a new server at 75% CPU on the repeaters, as indicated by the AWS CloudWatch monitors. In the case of a heavy influx of viewers, this gives the new instance enough time to get up and running before the other repeaters hit 100% CPU.  Wowza tells me this is at about 180Mbit/sec on a small instance, which for us means around 300 streams. The CPU threshold of 75% works out to about 260 streams.

Unfortunately for our online worshipers, our web server was bogging down pretty hard at

Web Server CPU usage

Web Server CPU usage

the beginning of the service, where the two CPU cores were maxed out for about 15-20 minutes, which translated into slower page loads. The database server wasn’t sweating too hard, so I suspect this could have been helped with better PHP caching. Fortunately for me, this had the effect of slowing down the rate of incoming streams, which allowed me to get new repeaters going before the existing ones started choking.

You can see in the graph where we added new repeaters, and how fast they ramped up. It also shows how incredibly well Wowza’s built-in load balancing works. We eventually leveled out at a little over 1100 streams, which meant our EC2 instances were cranking out 600-700 Mbps for nearly an hour:

AWS Network Usage

AWS Network Usage

Meanwhile, this is what we were seeing on Woopra (note the fortunate souls escaping the ice storm in Aruba and the Cayman Islands!):

2-21-10-AM

Next step is to define rules in Cloudwatch for automatically scaling. For that to work, I’m going to need to build my own Wowza AMI, since the current method of starting repeaters involves sending the instance a startup package from the client. I’ll need to build this configuration into the server for CloudWatch scaling to work properly.

Live Streaming on a Budget (Part 5) – Automating EC2

July 8, 2009 |   |  5 Comments  |  Internet Campus, streaming

I mentioned a few posts back that I was looking for a way to automate startup and shutdown of the servers. Thanks to some great sleuthing by Justin Moore at Granger Community Church, I got some scripts to start from. I had to make some modifications to suit our exact purposes, but that was relatively easy.

Ingredients:

Linux system with:

  • Java Runtime (in Ubuntu, the package is sun-java6-jre)
  • Amazon EC2 API Tools (installed in /opt/ec2/tools)
  • Wowza Startup Packages (installed in /root/ec2)
  • EC2 keys (installed in /root/ec2)

Note: Because these scripts are run from cron, you’ll need to put all your environment variables to run EC2 at the beginning of each one.

I have 6 separate versions of the startup and termination scripts, one for each server I need to start. I could roll it into one big script, but putting them in their own individual ones not only lets me do an individual machine manually, I can run them all in parallel from cron, which shortens the startup time.

The startup script functions as follows:

  1. Assign environment variables for EC2 and for the machine parameters
  2. Launch machine with ec2-run-instances, redirect output to a temporary file*
  3. Parse temporary file and extract the instance ID, and put it into an environment variable
  4. Write contents of instance ID environment variable to a file in /root/ec2 for use by the shutdown script
  5. Wait 30 seconds
  6. Start checking instance status every so we know when it’s running (wait 10 seconds to check again if it’s not)
  7. Attach any EBS volumes (Optional – I don’t currently need this, so it’s commented out)
  8. Assign Elastic IP
  9. Execute any additional commands via ssh (Optional, I don’t have any that need to run)

* The original scripts use /tmp/a, which is fine, but I had to make each script do its own temporary file since all 6 were running simultaneously and I ran into problems with getting the right Instance IDs set.

The shutdown script works like this:

  1. Query AWS for all running instances
  2. Issue EC2 termination call
  3. ???
  4. PROFIT!

Lastly, put it in your crontab:

# m h  dom mon dow   command
15 8 * * 0 /root/start-windows.sh
25 8 * * 0 /root/start-origin.sh
25 8 * * 0 /root/start-iphone.sh
25 8 * * 0 /root/start-repeater1.sh
25 8 * * 0 /root/start-repeater2.sh
25 8 * * 0 /root/start-repeater3.sh

0 19 * * 0 /root/term-windows.sh
0 19 * * 0 /root/term-iphone.sh
0 19 * * 0 /root/term-origin.sh
0 19 * * 0 /root/term-repeater1.sh
0 19 * * 0 /root/term-repeater2.sh
0 19 * * 0 /root/term-repeater3.sh

This starts up all of them (except the Windows instance which needs more time) at 8:25 on Sunday morning and shuts them down at 7 on Sunday evening.  (be sure that if you’re using GMT on your Linux box to take that into account).

If you’re using an encoder that can be started and stopped on a schedule, synchronize your times with this, and you’ll be golden. The Wowza EC2 images take about 60-90 seconds to fully get up and running, and the Windows one takes about 10-15 min. Currently, the Windows server pulls from the encoder via a 1:1 NAT rule, so the WME instance can be running before the EC2 server is going. When EC2 is ready, it simply connects to the encoder and is off and running.

Live Streaming on a Budget (Part 4) – How it works

July 5, 2009 |   |  Comment  |  Internet Campus, streaming

We started streaming to iPhones today. Huge success, way easier than it ought to be, now that the iPhone does HTTP streaming and Wowza’s V2 software supports everything needed to stream to the FruitFone. All we had to do was shell out $250 to MainConcept for their AAC encoder plugin for Flash Media Live Encoder. (Although we subsequently discovered that there is a bug with the MainConcept encoder that cause audio sync problems on iPhone, so we’ve moved iPhone encoding off to a separate machine)

There are a lot of layers to this onion, so I put together a block diagram that links everything from the cameras to client.

Live Streaming On a Budget (Part 1) – Genesis

June 3, 2009 |   |  4 Comments  |  Internet Campus, streaming

When our senior pastor started casting his vision of an Internet Campus which would revolve around a live (or nearly-live) stream of worship, it became pretty apparent that this was not going to scale well or cheaply. Over the course of the summer of 2008, we started exploring creative ways to do the impossible with nearly no money.

Read More