Enhancing the public Wi-Fi experience

Recently, there was an excellent blog post from WLAN Pros about “Rules for successful hotel wi-fi“. While it is aimed primarily at Wi-Fi in the hotel business (where there is an overabundance of Bad-Fi), many of the tips presented also apply to a wide variety of large-scale public venue wifi installations. Lots of great information in the post, and well worth a read.

At the 2016 WLPC there was an interesting TENTalk from Mike Liebovitz at Extreme Networks about the pop-up wifi at Super Bowl City in San Francisco, where analytics pointed to a significant portion of the traffic being headed to Apple.

Meanwhile, a few months later at the 2016 National Church IT Network conference, I heard a TENTalk about Apple’s MacOS Server, where I first heard about this incredibly useful feature (sadly, it wasn’t recorded, that I know of, so I can’t give credit…)

With most of the LPV installations I’ve worked on, I’ve found the typical client mix includes about 60% Apple devices (mostly iOS). For example, this is at a large church whose wireless network I installed. (Note that Windows machines make up less than 10% of the client mix on wifi!)

Client mix from Ruckus ZoneDirector

OK, So what?

This provides an opportunity to make the wifi experience even better for your (Apple-toting) guests. Whenever possible, as part of the “WiFi System” I will install an Apple Mac Mini loaded with MacOS Server. This allows me to turn on caching. This is not just plain old web caching like you would get with a proxy server such as Squid, but rather a cache for all things Apple. What does this do for your fruited guests? It speeds up the download of software distributed by Apple through the Internet. It caches all software and app updates, App Store purchases, iBook downloads, iTunes U downloads (apps and books purchases only), and Internet Recovery software that local Mac and iOS devices download.

Why is this of interest and importance? Let me give you an example: A few years ago, we were hosting a national Church IT Round Table conference at Resurrection on a day when Apple released major updates to MacOS, iOS, and their iWork suite. In addition to the 50 or so staff Mac machines on the network, there were another hundred or two Mac laptops and iThings among the conference attendees. The 200MB internet pipe melted almost instantly under the load of 250 devices each requesting 3-5GB of updates. That would have melted even a gigabit pipe, and probably given a 10Gbps pipe a solid run for its money (not to mention bogging down some of the uplinks on the internal network!. Having a caching server would have mitigated this. It didn’t do great things to the access points in the conference venue either, all of which were not only struggling for airtime, but also for backhaul.

Just by way of an example, Facebook updates their app every two weeks and its current incarnation (86.0, March 30, 2017) weighs in at 320MB (the previous one was about half that!), and its close pal Messenger clocked in at 261MB. Almost everyone has those to apps, so they’re going to find itself in your cache almost instantly, along with numerous other popular apps. Apple’s iWork suite apps and Microsoft Office apps all weigh in around 300-500MB apiece as well. This has potential to murder your network when you least expect it. (A few years back, the church where I was working hosted the national Church IT conference that happened to coincide with Apple’s release of OSX Mavericks, and a major iWork update for both iOS and MacOS. The conference Wi-Fi and the church’s 200Mbps WAN pipe melted under the onslaught of a couple hundred Apple devices belonging to the guest nerds and media staff dutifully downloading the updates.)

In any case, check out the network usage analytics from either your wireless controller or your firewall. If Apple.com is anywhere near the top of the list (or on it at all), you owe it to yourself and your guests to implement this type of solution.Network Statistics from Ubiquiti UniFi

The Technical Mumbo-Jumbo


As mentioned previously, a Mac Mini will do the job nicely. If you’re looking to do this on the cheap, it will happily run on a 2011-vintage Mini (you can find used Mac Minis on Craigslist or eBay all day long for cheap), just make sure you add some extra RAM and a storage drive that doesn’t suck (the stock 5400rpm spinning disks on the pre-2012 era Mac Mini and iMacs were terrible.) Fortunately, 2.5″ SSDs are pretty cheap these days. Newer Minis will have SSD baked in already.

If you’re wanting to put the Mac Mini in the datacenter, you might want to consider using a Sonnet RackMac Mini (which is available on Amazon for about $139) and can hold one or two machines.

Sonnet RackMac Mini

You can also happily run this off of one of the 2008-era “cheese grater” Mac Pros that has beefier processing and storage (and also fits in a rack, albeit not in the svelte 1U space the Sonnet box uses). If you have money to burn, then by all means use the “trash can” Mac Pro (Sonnet also makes a rack chassis for that model!).

This is a great opportunity to re-purpose some of those Macs sitting on the shelf after your users have upgraded to something faster and shinier.

Naturally, if you’re running a REALLY big guest network, you’ll want to look at something beefy, or a small farm of them Minis with SSD storage (the MacOS Server caching system makes it quite easy to deploy multiple machines to support the caching.)

The Software

MacOS Server (Mac App Store, $19.99)

Since most of your iOS guests will have updates turned on, one of the first things an iOS device does when it sees a big fat internet pipe that isn’t from a cell tower is check for app updates. If you have lots of guests, you will need to fortify your network against the onslaught of app update requests that will inevitably hit whenever you have lots of guests in the building.

The way it works is this: When an Apple device makes a request to the CDN, Apple looks at the IP you’re coming from and says, “You have a local server on your LAN, get your content from there, here’s its IP.” The result being that your Apple users will get their updates and whatnot at LAN speeds without thrashing your WAN pipe every time anyone pushes out a fat update to an app or the OS, which is then consumed by several hundred people using your guest wifi over the course of a week. You’ve effectively just added an edge node to Apple’s CDN within your network.

Content will get cached the first time a client requests it, and it does not need to completely download to the cache before starting to send it to the client. For that first request, it will perform just as if they were downloading it directly from Apple’s servers. If your server starts running low on disk space, the cache server will purge older content that hasn’t been used recently in order to maintain at least 25GB of free disk space.

MacOS Caching Server Configuration

The configuration

If you have multiple subnets and multiple external IPs that you want to do this for, you can either do multiple caching servers (they can share cache between them), or you can configure the Mini to listen on multiple VLANs:

Mac OS network preferences panel

Once you have the machine listening on multiple VLANs, you can tell the caching server which ones to pay attention to, and which public IPs. The Mac itself only needs Internet access from one of those subnets.

MacOS Server Caching Preferences

The first dropdown will give you the option of “All Networks”, “Only Local Subnets”, and “Only Some Networks”. Choosing the last one opens an additional properties box that allows you to define those networks:

Mac OS Server Cache Network Settings

The second one gives you the options of “Matching this server’s network” or “On other networks”. As with the first options, an additional properties box is displayed.

In both cases, hit the plus sign to create a network object:

Mac OS Server Create a New Network

It should be noted here that this only tells the server about existing networks, but it won’t actually create them on the network interface. You’ll still need to do that through the system network preferences mentioned previously. If you don’t want to have the server listen on multiple VLANs, you can just make sure its address is routable from the subnets you wish to have the cache server available, define the external and internal networks it provides service to, and you should be off to the races. This will provide caching for subnet A that NATs to the internet via public IP A, and B to B, and so on. Defining a range of external IPs also has you covered if you use NAT pooling.

There’s also some DNS SRV trickery that may need to happen depending on your environment. There are some additional caveats if your DNS servers are Active Directory read-only domain controllers. This post elaborates on it.


Is it working?

Click the stats link near the top left of the server management window. At the bottom is a dropdown where you can see your cache stats. The red bar shows bytes served from the origin, and green shows from the cache. If you only have one server doing this, you won’t see any blue bars, which are for cache from peer servers. Downside is that you can only go back 7 days.

On this graph, 3/28 was when there were both a major MacOS and iOS update released, hence the huge spike from the origin servers on Apple’s CDN. Nobody has updated from the network yet… But guest traffic at this site is pretty light during the week. I’ll update the image early next week.

MacOS Server Cache Stats

Other useful features

A side benefit of this is that you can also use this to provide a network recovery boot image on the network, in case someone’s OS install ate itself – on the newer Macs with no optical drive, this boots a recovery image from the internet by default. This requires some additional configuration, and the instructions to set up NetInstall are readily available with a quick Google search.

If you want, you can also make this machine the DHCP and local DNS server for your guest network. With some third-party applications, you can also serve up AirPrint to your wireless guests if they need it.


From a guest experience perspective, your guests see their updates downloading really fast and think your WiFi is awesome, and it’s shockingly easy to set up (the longest and most difficult part is probably the actual acquisition of the Mac Mini) It will even cache iCloud data (and encrypts it in the cache storage so nobody’s data is exposed). Even if you have a fat internet pipe, you should really consider doing this, as the transfers at LAN speed will reduce the amount of airtime consumed on the wireless and the overall load on your wireless network. (Side note, if you’re a Wireless ISP, this sort of setup is just the sort of thing you ought to put between your customer edge network and your IP transit)

Of course, you could also firewall off Apple iCloud and Updates instead, but why would you do that to your guests? Are you punishing them for something?

Android/Windows users: So sad, Google and Microsoft don’t give you this option (Although Microsoft sort of does in a corporate environment with WSUS, but it’s not nearly as easy to pull off, nor is it set up for casual and transient users). I would love it if Google would set up something like this for play store, Chromebook, etc, as about half of the client mix that isn’t from Apple is running on Android. You can sort of do it by installing a transparent proxy like squid.

Now, if only we could do the same for Netflix’s CDN. The bandwidth savings would be immense.


(Added November 16, 2017)

As of the release of MacOS High Sierra and MacOS Server 5.4 (release notes), the caching service is now integrated into the core of MacOS, so any Mac on the network can do it, without even needing to install Server. The new settings are under System Preferences > Sharing:



The video game is changing

Nope. Not talking about your XBox or Playstation or even your Wii.

A while back, I posted about why Blockbuster is screwed. The scene just got bleaker, and not just for Blockbuster. Now the entire Cable TV industry is facing a major conceptual shift.

Mark Melanson blogged today about Netflix mulling over the idea of ditching the physical media distribution concept that they perfected. Netflix has already induced a lot of insomnia with the senior management at Blockbuster. The cable people need to start worrying for two reasons:

  1. This is going to clobber Pay-Per-View revenues, especially if Netflix gets major licensing deals on fresh content.
  2. This is going to clobber the data networks that these same cable operators are selling to their TV customers.

But there’s more. The way we watch content in general, not just movies, is changing dramatically. What the cable companies fail to realize is that they’re not really in the content business. They’re in the business of selling a wire into your house, and they need to provide you with a compelling reason to pay them for that wire, so they piggyback a bunch of TV on it. In many cases, they’ll bundle IP and phone service too.

One of the problems is that when you’re selling a wire as a content delivery mechanism, you either have to produce a lot of compelling content, or acquire it somehow. There’s plenty of that out there to be had, but at a price. And that can lead to the content producers holding their customers hostage as a bargaining chip against the middleman. By the way, Fox and Cablevision, have you noticed that this makes your customers very angry? I bet Major League Baseball is selling a ton of online viewing subscriptions. That was revenue that could have been yours.

Fortunately, consumers have a few options to consume content that isn’t dependent on the company providing the wire into the house. One only has to look at the success of Hulu, Major League Baseball, and Netflix to see that. Of course if your internet access is coming from the same place as your TV, the content provider can quite easily lock you out, as Fox did to their Cablevision consumers.

The problem is, in the current environment, TV is still very much something tied to time and place. What content you get over cable or broadcast is subject to the scheduling whims and programming choices made by the stations, networks, and cable operators.

We as consumers have tried to work around this with DVRs (timeshifting) and devices like SlingBox (placeshifting) in order to consume content on our terms.

This works, but to a point. It also provides unecessary stress on the last mile of the networks. It’s also ridiculously expensive for the consumer. I no longer have cable. Or a TV, for that matter. Most everything I watch is picked up over-the-air by my Windows Media Center DVR and watched via another machine on my network, or online via Hulu or the content provider’s website.

The downside to this arrangement is that when watching online, there’s still a delay from the original airtime to when it’s actually made available on the web. This generally doesn’t bother me as I’m not a slave to TV schedules, but I do miss out somewhat on the shared experience of millions of others watching (and tweeting about) a show at the same time.

Then there are other shows that aren’t available in either format. I can’t watch Mythbusters on the web very easily without violating copyright law. There’s always Netflix and TV Show DVDs (which have been hugely successful) for that, but it’s not convenient.

Here’s what most of the content companies are failing to realize: Consumers will find a way to watch the content they want to watch, when they want to, on the device they want to, and generally care little about intellectual property laws meant to preserve originality.

If you’re a content company that’s not making your full content available via streaming, you’re missing out on a potential audience. It also has to be easier to consume legally than illegally.

Hulu is a great example of making it easy to consume content. Netflix is doing a great job of adapting.

The other great challenge with cable providers is that there’s a finite amount of content that can be stuffed down the wire. The current model involves sending everything down the wire at once and having the machine at the consumer’s end of the wire display a given one. Some great technological progress has been made to increase that capacity, but it’s still finite. Wouldn’t it be a lot simpler to send only the content actively being consumed down the wire?

Better still, give me a virtual DVR in the cloud and let me pick from a whole host of content. I still want to watch my favourite shows when I’m on the road. I can’t do that with cable. I may have eclectic tastes that don’t line up with what makes money for the cable operator. If I like to watch Curling and Cricket, I’m out of luck, because there may be 3 of us in the whole area who care about those sports.

Say you’re the Discovery Channel, or Fox. Instead of selling your content wholesale to the cable operators, stream your content directly to the consumer, in HD. I still think you can make money doing this, either with advertising or paywalls.

Imagine a virtual “cable” operator. Not bound by geography or cable plants, but rather open to the entire planet, and you offer a menu of content. Charge by the channel. Or by the show. (We’re talking micropayments here, but if most people are willing to shell out 60-100 bucks a month for a buffet of channels, and ultimately go back to the same 10 channels, there should be money to be made). You don’t even have to provide the streaming infrastructure, let the content providers worry about that. Just sell/broker access to it. The distribution is handled by the major CDNs anyway.You can even offer obscure content that doesn’t have a lot of demand. Stop being a slave to schedules. Sure, release new content every week, but let people watch it on their schedule. If you’re not sure how to make that work for you, go ask Felicia Day. She’s got it figured out.

The 2010 Winter Olympics were a good step in that direction. Even so, geographical restrictions on content (imposed mainly due to licensing issues) really got in the way. Many people found ways around it with proxies. Here’s a clue to content providers: Consumers don’t really care about geography. Why should I be disallowed from watching a show or event on the CBC or the BBC simply because of where I happen to live? Your content is compelling to me! I’m even willing to pay for it, either with real money or by watching your ads (just don’t get too crazy with the ads or I’ll go somewhere else). You’re missing out on a revenue opportunity when you should be going after every one of those you can get.

Suddenly, the guys in the business who are charging for a wire to the house should be getting nervous. The current cable paradigm is tantamount to charging $100 for a chinese buffet with only one steam table. The value proposition simply isn’t there. That fact that you’re still in business at all is a testament to the power of monopolies and heavy-handed legal action.

Cable operators need to get out of the content business. It’s killing them. Might as well get out of the voice business too, since that’s not going to stick around long. But if you’re willing to take that wire and provide me with a transport mechanism for all this content out there (in other words, IP access), I’m all over it. I’m a customer of my cable company. And all I buy from them is data. I’m fortunate enough to have cable competition in my area, but the competitor wants to charge me extra for not having TV content clogging up my wire. Sorry, that doesn’t fly with me.

Why on earth would you want to restrict the size of your audience? There are millions of consumers wanting to consume a ton of available content out there. Don’t get in the way. If you do, the consumers will cut you out of the action and you’ll eventually find yourself off in the booth in the corner with the magazines and newspapers, crying into your beer and wondering why nobody loves you anymore.

Update (10/23/10): CBS, ABC, and NBC demonstrate that they don’t get it. They are shutting out Google TV users from viewing their content. Oh well, they’ll figure it out eventually. If they’re lucky, before they become completely irrelevant.

Update 2 (10/29/10): And now we hear that XBox Live is now bigger than Comcast.