Aruba AP Provisioning

As part of trying to wrap my own head around the various profile dependencies in actually provisioning an Aruba AP , I’ve mapped it out. This is the <stuff> that goes into this process:

provision-ap
read-bootinfo {wired-mac|ip-addr|ap-name} <data>
<stuff>
reprovision {serial|wired-mac|ip-addr|ap-name} <data>

As you go to provision an AP, start on the outside of this map and work your way in. This will make sure that all the various profiles you need are in place. The web UI hides some of this stuff from you and doesn’t organize it as logically as one might expect.

When doing this on the CLI in Mobility Master Conductor, make sure you’re in the right corner of your hierarchy (namely, /md or /md/GROUP). And remember that on MMMCR, show run is not nearly as useful as show config effective… And config purge-pending sure comes in handy when you goof something up.

You can also do show profile-hierarchy but that only shows the profile entries… And it doesn’t fit neatly in a terminal window…

Caveat: This is not comprehensive by any stretch. There are dozens more options, these are just the more common ones. If I goofed, let me know. All the gory details can be found in the ArubaOS User Guide.

Gear Review: GoFanco Wireless HDMI

A few weeks ago, I went to Amazon and picked up a cheap wireless HDMI transmitter to solve a camera connection challenge at the church. I needed to send a GoPro feed back to the booth without running cables all over the floor (or worrying about the GoPro’s live streaming latency — it uses HLS with a real short segment size– and getting that stream into the switcher was not a trivial task).

As is so often the case with these “off-brand” (or less well-known brand) devices, my expectations were low, and I fully expected to return it after a week.

That didn’t happen. Not only was it easy to set up, the picture quality was excellent, and latency almost nonexistent. I immediately picked up another set to run a mobile confidence monitor cart. It’s also able to send IR control to the receiver.

As it turns out, GoFanco is starting to make a name for themselves in the video accessories market for both pros and consumers, not entirely unlike how BlackMagic got their start. They offer quite a wide assortment of gizmos to move video signals around.

The mobile GoPro rig. This could also potentially be used for linking live UAS footage back to a switcher.

Since I am a wireless network engineer by profession, I had significant concerns about how well this would behave in the spectrum – it advertises that it uses 5GHz, and I expected it to grab as much spectrum as it could (as most wireless video devices tend to), and walk all over everything else in the band. So I hauled out my Ekahau Sidekick and its spectrum analyzer to see how well-behaved it would be… And I was pleasantly surprised to discover that it was very well-behaved on the Wi-Fi spectrum… because it’s actually running Wi-Fi!

It’s running 802.11ac on a 20MHz channel (and the channel selection allows 10 different channels, which tells me it’s running on UNII-1 and UNII-3 and avoiding the DFS bands). Airtime usage is quite efficient, around 4%, which is shocking for a video application. And perhaps most useful is that it runs on 5VDC, and the supply is rated at 2A… Which means I can use a USB battery to power the transmitter and the GoPro (and a 20Ah slice will run this rig All. Day. Long.

Additional features allow not just 1:1 link, but 1TX:2RX, 2TX:1RX, all using a single channel. And because it’s quite efficient in spectrum/airtime usage, it does this in such a way that will coexist peacefully with your Wi-Fi.

It also means that if a presenter brings a laptop and wants to put it up on the screen, the transmitter can be powered from the laptop itself. This thing is a definitely a worthwhile addition to your tool kit.

The gory technical details,

  • Channel 0: Wifi Channel 36 (20MHz) (Default)
  • Channel 1: Wifi Channel 44 (20MHz)
  • Channel 2: Wifi Channel 157 (20MHz)
  • Channel 3: Wifi Channel 157 (40MHz)
  • Channel 4: Wifi Channel 149 (20MHz)
  • Channel 5: Wifi Channel 153 (20MHz)
  • Channel 6: Wifi Channel 149 (40MHz)
  • Channel 7: Wifi Channel 153 (40MHz)
  • Channel 8: Wifi Channel 165 (20MHz)
  • Channel 9: Wifi Channel 161 (20MHz)

This appears to be fairly smart frequency selection behaviour since these preprogrammed channels look like it will never set itself up on the secondary channel of a 40MHz pair, which is good for co-existence with other Wi-Fi. When powering up the unit, it will start on 0 but then switch to the last channel it was using once it completes booting, which only takes about 5 seconds.

Each channel is its own encrypted SSID named LK_<Channel #>.

There is a pair of LEDs on each unit: The transmitter has one that indicates it is getting a good HDMI signal, the other indicates that it has synced up with the receiver. On the receiver, one indicates that it has a good wireless signal (solid indicates a connection, blinking indicates active data transmission), and the other indicates that it has synced up with the transmitter. A mild annoyance here is that changing the channel on the receiver will not trigger the transmitter to switch channels. However, the included IR remote will let you do so. The transmitter also has an HDMI pass through, so you can insert it between a source and a monitor.

Here’s about a minute of traffic captured from the Sidekick. A channel change happens around 30 seconds in. The channel change process is pretty straightforward and exactly what you’d expect. When initiated from the receiver, it sends a deauth frame, and then the transmitter continues to beacon on its existing channel. When channel change is initiated on the transmitter, it will send a series of broadcast deauth frames to the SSID, change to the new SSID and start beaconing (this takes less than a second). Meanwhile, the receiver is looking for beacons from its pal, and when it sees the right SSID on its channel, it sends a broadcast probe request, gets the response from the transmitter, and goes through the standard association process. Management frames do not appear to be protected, so this device is vulnerable to deauth spoofing.

Data rate hovers around 100Mbps according to AristaPackets analysis of the capture. Given their use of off-the-shelf Wi-Fi for the networking component, I wouldn’t be surprised in the least to discover that the video protocol running underneath the hood was NDI, or something based on it. Why reinvent the wheel? I’d really love to crack open the encryption on this guy and see…

Given that this is using standard 802.11, the advertised range is about 50 metres, but it could easily be made to go longer distances simply by attaching a 2×2 MIMO directional antenna. Antenna connectors are RP-SMA.

One caveat: When I first set it up, the receiver was having a hard time staying up and maintaining signal… I quickly discovered that I had grabbed the wrong 5V power supply, and it was only able to source up to 1A – This device definitely needs more juice than that. Once I grabbed the correct 5V power supply, everything worked great. If you use a USB cable to power it, make sure the USB supply can source the full 2A (any supply designed for tablets or higher end smart phones should be adequate)

All in all, not a bad little setup for $200 and small change. It appears to be engineered above its price point, which is a great value.

“It’s ALWAYS DNS (or DHCP)”

There’s a common saying among my network engineering peers: “It’s ALWAYS DNS!”. For those not familiar with the concept, this refers to the alarming regularity with which networking troubles end up being caused by something trivial, such as name resolution. And when it’s not DNS, it’s usually DHCP. Those two troublemakers alone are responsible for some ridiculously large percentage of network support issues. (At least until someone at a tier 1 provider inserts a typo into a route table advertised to half the internet via BGP, and takes everything down, but I digress.)

Last weekend, I rebuilt my home wireless network from an Aruba Instant cluster back to a controller based network, using ClearPass as an authentication and authorization backend for the home network. Gross overkill for a home network, but it gives me stick time on stuff that I need to know for work, at a much grander scale.

But first, a little background into the Aruba Way of doing things: In an Instant cluster, the wireless networks are bridged to a VLAN that is trunked to the access point. You can also do this with campus networking, but managing all those VLANs on every port that feeds an access point is usually a recipe for forgetting something vital. So the campus model lets you build a single access VLAN on your AP ports, and the AP establishes a GRE tunnel back to the controller cluster (which also allows for some great redundancy and high availability options), and the various VLANs terminate on the user anchor controllers (because each user has their own tunnel back to the controller, which allows you to segment their traffic out and handle it at layers 4-7 based on a variety of rules, and the only thing going over the wire is an encrypted tunnel, which is a significantly better security posture should someone unethically decide to monitor traffic on a switch port when they shouldn’t.

This is also where ClearPass comes into play – How user sessions and traffic are handled is defined in roles. Each role consists of various rules. How roles are applied are defined by policies. You can map roles to users and/or machines with the magic of ClearPass, and then when someone connects to the wireless network, ClearPass can return a role (and it can map a different role based on whether you authenticated with a username/password, a certificate, or any one of a number of other data inputs). Basically, when ClearPass returns the OK to the controller, it also includes a bunch of attributes for that user, including roles. It’s extremely powerful magic, and when wielded wrong, it can cause no end of heartache trying to figure out just what exactly went haywire. And I’m still very much a ClearPass n00b.

Which brings me back to my newly built and ClearPass-enabled network. And so like every good story…

No $#!+, there I was…

When I connected, it would take a good 10 minutes before I could access the internet. And so, I’m wondering what I screwed up in my ClearPass setup that would have done this… But the roles were being assigned correctly, and the rules associated with those roles were pretty straightforward: “allow all”. So why in the heck were devices on the home network taking forever and a week to get an address? This was not happening on my IoT and guest networks.

First, I realized that my devices were associating just fine, so ClearPass and the role derivation were working correctly, which immediately acquitted the Wi-Fi (but as far as the others in my house were concerned, the Wi-Fi was still screwed up). But that meant I had a good Layer 2 connection. I tried to make sure that the VLAN was properly connected from the pfSense router to the core switch, and the controllers (running in VMWare) were properly trunking to the distributed vSwitch and also out to the core switch. Everything on that front looked good. I tried manually assigning IPs to the wireless clients on the home LAN, and they worked great. So L3 worked, which implied L2 did as well. And when clients on the home network did eventually get an IP address, they worked fine as well. So nothing was being bottlenecked anywhere either (I should hope not, as the VMWare hosts and the router are all connected to the core switch with dual 10-gigabit fiber links!).

After a few days of racking my brain over this, and hearing the people who live in my house continue to complain about network weirdness (thankfully, my family is not doing virtual school/work… except for me), I finally resigned myself to doing what I should have done in the first place: Breaking out Wireshark and figure out just what was actually happening on the network. DHCP is pretty simple, so finding out what broke should be straightforward, right?

Quick refresher on DHCP: The process of obtaining a DHCP address goes like this:

Since I knew I had good L2 connectivity, I fired up Wireshark on my laptop, capturing what was going on at L2, and would move to other points in the network if I needed to. The first thing I saw is that a residential network, even with isolated guest and IoT traffic, while nobody else in the house is using it, is a fairly chatty place. I saw a bunch of multicast traffic (I have a lot of Apple devices), even IP broadcast traffic. And there, among all that, was the DHCP process. Discover. Discover. Discover. Offer. Request. Request. Request. Discover. Discover. Offer. Request. Request. Request. Discover. Discover. Discover. Offer. Request. Request. Request. The more astute among you may have noticed something missing from this sequence. Something rather… important.

Turns out, my DHCP server was making an offer, and then ghosting my devices as soon as they responded to that offer. And periodically, a DHCP ACK would sneak through. And by now, it had started happening on my IoT network as well, as half my Nest Protect alarms were now showing offline. But that told me one very important thing: that my DHCP server was in fact online, reachable, and responding. Up until that very last point.

So I then did what any sane engineer would do:

I had already restarted the dhcpd on my pfSense box, so I didn’t have much faith in the curative effects of a digital boot to the head, but what the heck, can’t hurt, right?

And that’s when I saw it. I went down to my lab, and there, on the front of the DL360 that is running my router, is an angry orange light which should normally be a happy little green. Uh-oh.

So, I pop out the handy little SID tray, to see what it’s angry about… And this is not something a server admin wants to see:

Yep, that’s flagging all three memory modules in Processor 1’s Bank A. This just became more than a simple reboot. Sure enough, when it went through POST, it flagged all three modules. Power off, slide out the server (rails FTW), and perform that tried and true troubleshooting method I learned and perfected in the Air Force a quarter century ago: Swaptronics. Move a suspected bad component and see if the problem follows. So, I switched all the DIMMs from bank A with those in Bank B. If the fault stayed with Bank A, then I had a bum system board. If the fault followed the DIMMs to Bank B, then the fault was in the DIMMs. I really wanted the fault to follow the DIMMs.

Plug it all back in, and fire it up, and the fault was…

NOW IN BANK B!!!! Hallelujah, I don’t have a bad server on my hands!

So now I shut it down, tossed the bad DIMMs in the recycling bin (yes, our recycling pickup actually takes e-waste, which is really nice when you’re a nerd with way too many electronic bits), and repopulated/balanced the banks (I also had to remove a fourth DIMM to keep things even, but it’s a known good part, so it did not go to recycling).

I fire the machine back up, and yay, it’s no longer grumpy about the bad memory, although it is briefly perplexed by the fact that it now only has 24GB instead of 32GB, and has somehow realized that it just had a partial lobotomy. After a few minutes of much more intensive self-testing than usual, it boots up pfSense, and gives me the happy beeps that pfSense does when it’s fully booted (for those of us who run our pfSense boxes headless!)

The moment of truth: I connect my laptop to the Wi-Fi (with the wireshark still circling)… and sure enough, the DHCP ACK comes through on the first try… So as near as I can tell, whatever part of the system RAM contained the bit of code required to send the DHCP ACK had suffered some kind of stroke, but not one severe enough to take the whole box or even the operating system down.

See? It’s always DHCP.

EDIT: Turns out there was also more to this – Wired clients (and access points) started getting DHCP right away after fixing this, but wireless was still giving me fits. As it turned out, There was something about the Aruba mobility controllers terminating user sessions that played havoc with the hashing algorithms that VMWare uses to handle NIC teaming on switch uplinks, and the ACKs were coming back through a different path and getting lost along the way.

For the moment, I disabled one of the 10G links to the switch until I can figure out what magic incantations I need to make on the vSwitch to get the hashing algorithms to properly use the multiple connections with the VMCs – or I may just use the second 10G interfaces for vMotion or something.

and that, kids, is how I used Wireshark to diagnose a system memory problem.

Hands On : Aruba Instant

After our quick little tour of Aruba InstantON, I’m going to move up to the next level of Aruba gear: Instant.

The naming can be a little confusing to the ArubaNoob, but Instant has been part of Aruba’s product offering for a very long time. While it appears controllerless, it still makes use of a virtual controller that lives inside the APs on the network (and in case the AP running the controller goes offline, the remaining APs on the network decide on a new leader by holding a rap battle or a dance-off. OK, just kidding. They actually do a sort of digital version of Rock, Paper, Scissors, Lizard, Spock.

This virtual controller concept has also been done by Ruckus with their Unleashed platform, which in terms of functionality is somewhere between Instant and InstantON, and Cisco’s Mobility Express. I’m not 100% sure, but I think Aruba had it first.

In previous generations of Aruba access points, you either purchased an Instant AP (IAP), a Campus AP (CAP) , or a Remote AP (RAP). The latter two required a Mobility Controller (MC). You definitely couldn’t RAP without an MC. Now, all APs ship as Universal APs and figure out which mode to be when they boot up, and can be easily converted from one to the other (in the dog park that is Ruckus Unleashed, you would have to reimage the AP with new firmware).

Who it’s meant for

Instant is designed for small and medium business environments, and home labs of geeks who subscribe to the idea of “if it’s worth doing, it’s worth overdoing” (My home wireless network right now consists of 7 APs in an Instant cluster). It also is very useful in large enterprises that consist of many small locations, especially once you start managing them all with Central. If you have a chain of coffee shops or boutiques that only require a few APs, then Instant+Central is definitely something you should look at. If you only have one, InstantON is more your speed.

Instant does not require any per-AP licensing, but it still includes a lot of the features you find on the campus systems. It even includes an internal RADIUS server and user database so you can do enterprise authentication (as of 8.7 which was just released in July 2020, you can even do up to 24 unique passphrases with MPSK before having to get ClearPass involved, which is real handy for IoT networks that use crappy chipsets that don’t support enterprise auth). It will also do an internal captive portal. It still has role-based access control, which provides layer 3 policy enforcement at the AP, including content filtering. And much like the InstantON APs can do, you can even use an Instant AP as your internet gateway (guess where InstantON learned it from?). You can even use it with ClearPass and all the goodies that come with that.

When a Universal AP powers up, it goes through the following process:

If setup mode is not accessed within a period of 15 minutes, the UAP reboots and goes through the process again. It can be a lonely existence. (this mode is not unusual to find in large campus networks where there exists a network disconnect at Layer 2 or Layer 3 between the AP and the controller. Chasing these down on a cruise ship is maddening… but it gets you a lot of steps.)

Setup Mode

Once the AP is in setup mode, it will broadcast an open SSID called SetMeUp-DD:BE:EF (where the last half is the last half of the wired MAC address of the AP). Connecting to this SSID will bring you to the configuration page (it will even conveniently pop it up in the captive portal window if your OS has such a thing). You can also access this by opening a browser to https://setmeup.arubanetworks.com, which it looks up via mDNS. (Caveat: This doesn’t work so great if the AP does not have an uplink and an IP address on the network, even if that IP is not routable… And accessing it via IP address only redirects to the hostname, and mDNS doesn’t really like not having a network to do its thing. So give it an uplink, even if it’s just a WLANpi.)

I once was traveling through a midwestern airport where I was scanning the wifi (it’s a wifi nerd thing) when I saw a lone AP broadcasting “Instant” (which is what Instant used to do before AOS 8.x). I eventually found the AP in a restaurant, where it was sitting all by itself on the ceiling, still in setup mode with the defaults… A quick peek into the setup page showed that this thing had never been configured… I found the manager to let them know that someone didn’t finish a job they were likely paid handsomely for, and she told me it had been there for almost 3 years and nobody had any idea what it was for or remembered who installed it or when. The airport’s installed public system was Meraki.

Once you’re in the setup interface, you can then configure it to your heart’s content. Then, when you bring up a second and subsequent access points on the network, they will find the first one, grab the configuration, and join the party. This scales surprisingly well – you can run several dozen access points on a network like this (There’s no actual hard limit, and it’s been officially tested up to 128 APs, but this is definitely not recommended – that’s well into Campus AP territory). It may not be truly instantaneous (we do love instant gratification), but it’s pretty darn close.

Limitations

There are a few limitations to this mode of operation, in addition to the aforementioned scaling issues (if you’re used to a SOHO/SMB system like Ubiquiti, 100 APs will sound like a lot to you. Once you get into controller based networks with Aruba, even a thousand APs is middle of the road – I routinely work with networks well in excess of this).

A few of the things you can’t do with Instant:

  • AP Groups
  • AirMatch (Instant uses the older ARM techniques for RF management)
  • Tunneling to controller (yet…)
  • I’m probably forgetting some things…

Perhaps the most useful aspect of Instant is that it can either be managed in the cloud with Aruba Central (if you’re used to Meraki, you’ll love Central), or if your network requirements grow to where you need to get a controller involved, switching the APs over to that mode is quick and easy, and you don’t have to buy new gear.

Labbing It Up

If you want to play around with Instant, it’s pretty easy: Buy an AP. Or more. If you have to fund your own lab gear, there’s a ton of used and refurbished Aruba gear on Amazon or eBay (If you go with HPE Renew, you still get HPE’s legendary lifetime warranty on network equipment). Recently, I saw a whole bunch of Renewed AP-345s on ebay for under $200. Just make sure you get the correct country code (US or RW) – the two can’t coexist on the same Instant cluster (in a controller environment, the controller country code takes over and ignores the AP setting).

If you’re new to the Aruba product line, here’s a quick cheat sheet to figure out what kind of AP you’re getting. It’s not 100% exact, but it should give you a general idea of what you should be getting.

The first digit of the 3-digit model number indicates product generation:

  • AP-0XX (or just AP-XX): 802.11g
  • AP-1XX: 802.11n
  • AP-2XX: 802.11ac Wave 1
  • AP-3XX: 802.11ac Wave 2 with integrated BLE
  • AP-5XX: 802.11ax with integrated BLE and ZigBee

The second digit indicates capabilities (1XX series and up)

  • AP-X0X: 2 spatial streams
  • AP-X1X: 3 spatial streams (although the 51X series is 2SS on 2.4GHz and 4SS on 5GHz)
  • AP-X2X: 3 spatial streams, second Ethernet port
  • AP-X3X: 4 spatial streams, SmartRate port, Gigabit Port
  • AP-X4X: 4 spatial streams, dual SmartRate ports, dual-5GHz,
  • AP-X5X: 8 spatial streams, three radios (only AP-555 for now… that thing is a monster)
  • AP-X6X: Outdoor AP with 2 Spatial streams
  • AP-X7X: Outdoor AP with 4 spatial streams
  • AP-X8X: Outdoor AP with 60GHz (only AP-387)

The last digit indicates the antenna type. Odd numbers are internal, even numbers are external.

  • AP-XX3: Internal Omni
  • AP-XX4: Connectorized
  • AP-XX5: Internal Omni
  • AP-XX7: Internal Directional
  • AP-XX8: Connectorized and ruggedized,

APs with the H suffix indicate a wallplate mount designed for the hospitality industry. These APs also have a built-in switch. I love these APs.

Naturally, if you want to get the gory details, head on over to Aruba and look for the data sheet.

Stay tuned for the next Hands On post in which I will discuss Aruba Central.

Disclaimer: Aruba is my employer, but this post reflects my personal experience as a wi-fi nerd with Aruba products. Some APs were purchased on the open market, some were provided to me by my employer for lab use. This is not a paid promotion, and is not official Aruba communication. I am not part of the Instant product team.

Video Production Suite

I’m going to veer off from my usual diet of wireless posts to bring you a howto post on the new video production desk that I built for the Family Life Center at our church.

Over the past year, we’ve been putting together a major upgrade to the video equipment in the FLC, because this is what we had…

Tech Booth with 1980s vintage video equipment

If you’re young, you may not recognize some of this. That switcher is a Panasonic AG-MX50, coupled with a 3-camera PTZ system of the same era. The AG-MX50 was introduced sometime around 1990, and quickly gained popularity not just for live production, but for anyone doing tape to tape edits – I first used this same system when working in the A/V department in college, and it was a pretty slick piece of kit then. Our FLC was built in 2002, and eventually inherited this video system from the sanctuary when it was upgraded several years back. To say that this system is past its prime would be a gross understatement.

The projectors in the room were also getting on in years, and in dire need of replacement. So we looked at what we needed to bring the video in our most attended venue into the 21st Century, and continuing a video ministry that has been in operation since the mid-1970s.

Background: We are a United Methodist church in a county seat town in central Kansas – about 250 a week in worship between two venues – while that’s big by United Methodist standards, we’re not one of these big high-tech churches – we do a lot with a little. (Long-time readers of the blog will remember that I worked at one of those big high-tech churches about a decade ago… It’s remarkable how much of those ideas can be scaled down to a “normal” sized church. The overall project budget for this video upgrade was under $40,000, and we included in that about 20-25% wiggle room for random cables, connectivity, and, apparently, building furniture. That little stuff adds up after a while.

The first piece was to add a couple of 75″ TVs (Samsung 7 series) in the front of the room on either side of the stage. This allowed people sitting in front to see what was on the screens, which were roughly even with the first row of seating. Our existing projectors were running 4:3 off a VGA signal that was being split and amplified and sent over some heavy duty cables to the projectors. The TVs didn’t support any kind of analog input, so I ran some Belden 9116 RG6 cable and put a Decimator MD-HX on the computer end, and a couple of BlackMagic HDMI converters at the TVs, and a WiiStar VGA converter on the projectors, as well as an SDI to analog converter for the switcher running the stream. Later, when we added the new projectors (Panasonic PT-VMZ60), I put a Decimator MD-LX to get the signal to HDMI. the Projectors and TVs were daisy-chained on the SDI link.

The next piece of the puzzle was to upgrade our switcher – We started with a BlackMagic ATEM TV Studio, and added a couple of WiiStar analog to SDI converters, but the ATEM TVS is rather finicky about input formats, which only one of the converters managed to output correctly. So we ran the SDI from the Decimator into the TVS, and then used an analog to SDI converter to get the output of the analog switcher into the TVS as well, and switched using both. This is also when we added an Aja Helo encoder to the mix, as my Teradek VidiU (a demo from the vendor) had decided to call it quits.

After several months of this, the funds were finally available to purchase the Ross Carbonite Solo switcher we had decided upon (cheaper than the ATEM 1ME unit, with more features, but most importantly, it had 6 build in scan converters, meaning I wouldn’t have to get a small platoon of Decimator MD-HX units just to make the ATEM happy. We opted for the 9-input unit rather than the 13, as this was plenty for our needs. We also purchased the AW-RP150 camera controller so we’d be ready when the funds came in for the new cameras – We had to replace a lot of infrastructure in order to be ready for the cameras.

Naturally, this meant we were going to have to make a major update to our workspace – as you can see above, it was kind of cluttered and not very functional. So I set about making a custom desk that would accommodate all the gear, keeping cable messes out of sight, and making the overall system much more user-friendly.

I then considered the overall layout and found that a 50″ TV as a central multiviewer with two 24″ monitors stacked on either side would give us an array of displays that would give us a good overview of the production. I had initially planned on mounting it about 8″ above the surface of the desk, but that ended up being a lot less ergonomic than we had figured, so I ended up moving them down almost flush with the flat surface of the desk, meaning I had to mount the sound bar above (suboptimal, but functional).

A rough early layout.

I had about 8 feet of space to work with, and could go about 3 feet deep. After tinkering around in Visio, I was able to come up with a design that could be cut from two sheets of cabinet grade plywood, provide a sloped front, in which the switcher and camera controller could be flush mounted, while also providing four bays with 4U of rack space each, where I could mount things like power strips, network switch, a patch panel, and various other bits.

Cut sheet for the first sheet

Note: the vertical dotted lines on the ribs are NOT CUTS! the lumber yard missed that memo, and I had to work around that.

Cuts for the second sheet of plywood

An artifact of this being a two-sheet design means that there’s bar at the top of the sloped face that is fixed in place – That does somewhat impede the access to the top rack space (more on that in a second) – you could use a third sheet and make the movable panels go the whole height.

This is also where I discovered that the “imported birch faced cabinet plywood” that my local lumber yard sells is basically 3/4″ worth of layers of cheap luaun with a birch veneer and a whole lot of glue. I would not recommend this material or use it again. It’s a pain in the ass to saw and sand, and it splinters like crazy. It got the job done, but it’s got some challenges (running a hole saw on this stuff sucks). Well worth spending an extra few bucks on actual hardwood cabinet plywood.

Once I got the big cuts done by the lumber yard, I had another member of the tech team handle the angle cuts with his table saw (where he also noticed that the wood quality was poor), since I lacked that particular bit of equipment.

Pile of boards ready for assembly
Ready to start putting together…

Getting down into the details of the assembly required some planning ahead. I had initially planned for 6″ space inside, until I realized that our current presentation computer is 6.2″ high. Then I remembered that 4U of rack space is 7″, so I altered the design (before cutting!) to create four bays of 4U each, and ordered four sets of rack mounting rails to go in the space.

Working design for the assembly – there are three bays: two with a pair of 19″ rack bays, and one on the end that is a few inches short – Things like power supplies and a Mac Mini live in here. Because of the updated monitor placement, the Qu-PAC ended up inside.

You’ll notice here that I have a bunch of passthru holes and outlet packs on the back deck – those have mostly gone away owing to the TVs and screens taking up the back 5″ or so of the deck. I may still add a few on the back deck between the rack bays. Once I get the hinged lids going, they will contain a wireless charging pad for mobile devices.

The side view – Holes between the ribs allow for airflow and cable passthrough.
Assembling the unit – using a piece of rack equipment to get the spacing right. Holes in the back are for cable and ventilation. They were carefully placed so as not to encounter a stud when installed.
Making some headway.

When it came time to haul it over to the church, I removed the top panels just to make it easier to maneuver and carry into the booth.

Installation time: A couple of scraps from cutting the ventilation holes were squared off on the radial arm saw and I used them as cleats to support the desk (using a laser to align it with the top of the legs)

We got it within about 1/64″ of level… I’ll take it.
Got it at least somewhat functional for Sunday… We thought we had a few weeks to do this before we re-opened in-person worship, but then our council accelerated the timeline.

I left the front lip open to provide for airflow.

First sunday with the new desk. Lid is off as we figure out best arrangement and connections of the equipment. Once we get that all figured out, we’ll clean it up real nice.

I was able to find a set of matching Asus monitors that were inexpensive and used a VESA mount, and while those were on their way, I painted the wall behind them in flat stage black (makes the displays blend in better, and the wall was in dire need of a coat of paint anyway, so I used some of the paint left over from painting the stage)

Aligning the monitors required a lot of careful planning and the laser level. Here you can see the cable passthroughs and the fan packs.
Final layout without the lids – Still not 100% sure they’ll ever make it on… We shall see. Sound bar had to go up high (suboptimal) to keep the displays lower. It works OK for what we need it to though.

Materials List:

  • 2 or 3 sheets of 3/4″ cabinet grade hardwood plywood
  • 2″ finish screws (recommend Torx or Robertson head)
  • 4 sets of 4U rack brackets
  • 2 36″ piano hinges
  • 1 24″ piano hinge (you’ll need to cut this one to length)
  • Any trim or edge veneer you wish – We’re putting a lip on the bottom edge of the lid.
  • 2 AC Infinity AirPlate fan packs – Based on the way I built our desk, I had to flip the fans around to change the flow direction. You can either vent below the cabinet into the existing space, or vent to the rear (in our facility, the wall leads to a big empty ceiling space above the kitchen. These have a manual speed control, but a thermostat control version is also available.

Optional Components:

Tools:

  • Hole saws for the large holes
  • Forstner bits for smaller holes
  • Drill (and small bits for pilot holes)
  • Kreg Jig for pocket screws
  • Belt Sander with coarse belt (for evening things out)
  • Laser level (for installation)
  • Orbital sander with 120/180/220 grit pads (for smoothing things out)
  • Power driver (impact or drill – cordless helps a LOT here)
  • T square/Try square
  • Clamps
  • Table Saw
  • Radial Arm or sliding miter saw
  • Jig saw or reciprocating saw

Tech Equipment:

And that’s the story so far… I’ll update the post as changes warrant.

Tag, You’re It!

Cover Image: Unmasked (detail), (Brian Wall, 2014)

Just this past week, Ekahau released the latest iteration of their excellent wireless network planning software, and with this version, they’ve added a few features that many of us have been wanting for quite some time. Of course, we always want more, and there’s only so much the elves at Ekahau can do! So this leaves us with building our own tools to extract the data we need out of the project file. (Hey, Ekahau, you know what would be really awesome? an SDK for doing this!)

Fortunately, Ekahau has been really good about building a standards-based project file format (and not encrypting it or doing things that make it a pain to use your own data). Since the Ekahau software is built in Java (cross platform on Windows/Mac!), it’s logical for the data file to be in something like XML or JSON, and they have chosen the latter, and have effectively built a relational database in JSON, and bundled the whole thing up into a convenient zip file. It’s almost like they understand that their core market is made up almost entirely of customers who like to tinker with things.

Disclaimers:

Naturally, manipulating this file is something to be done entirely at your own risk, and if you break it, don’t go crying to Ekahau, because they don’t support mucking with their data file outside of their application (nor should they be expected to!) Make sure you have backups, etc, etc.

Also, this post is in no way based on any inside information from Ekahau, nor is it anything official from them – this is simply an analysis of the contents of the project file that anyone could do, whose nature as a zipped file full of JSON has been known for quite some time.

“I’m gonna get some tags… This is f’ing awesome”

Probably the coolest new feature in v 10.2 is the ability to add key:value tags to stuff. You can apply these tags to APs, either just the tag by itself, or a tag with a value associated with it. The Quick Select also lets you select any APs that have a particular tag key (although somehow they missed the ability to refine based on tag value, which I hope will be corrected in the near future).

Why is this useful? This allows you to add free-form information to access points, whether simulated or measured, that allows Ekahau to be more than just an RF simulation tool, and extends it into a full blown planning and deployment tool. Tagged information can be any kind of metadata you wish. things like:

  • Mounting hardware
  • Wired MAC address
  • AP Group
  • Serial Number
  • Zone
  • Switch
  • Port
  • Cable
  • IDF
  • … and the list is nearly endless.

This is in addition to the already rich metadata that is associated with the AP that are directly relevant to the RF modeling, such as mounting height, mounting surface, antenna angles, power, channel, antenna types, and so forth.

So how does it work? Pretty simple: on an AP, simply open the sushi menu at the top right, select “Tag AP”. You can also do this from the Edit AP or bulk edit screen when doing multiple APs. This will give you a list of existing tag keys already associated with the project (as well as tags you’ve used before on other projects), along with a free form box to enter your own, or add a value.

As of right now, there’s not a whole lot you can do within the Ekahau software once you have those tags (I would LOVE a table view of my APs and all the metadata, as well as ability to import/export to CSV or Excel), nor is template-based reporting on those tags documented at this point (although I expect they’re working diligently to document this). One key weakness of the template reporting system is that it all has to go through Microsoft Word (with a whole bunch of limitations imposed by that format), and that gets really hairy once you start creating data tables, especially if you want them in Excel or something else.

Side note: Using Excel as a database is really a terrible use of a spreadsheet, but it happens all. the. time.

Which brings me to manipulating/extracting your data by building your own tools. Several people have been doing this unofficially for years, but Ekahau doesn’t offer anything for this… yet.

I mentioned earlier that Ekahau’s data is stored mostly in JSON, which makes it really easy to work with using Python (or, for that matter, Java or perl if you’re into self-flagellation). Every object in the data file has an ID that ties it back to other objects. And that’s the key thing (literally). There are about 2 dozen separate files that track various data, and that’s how they all tie together. Notes and tag keys are each kept in their own file, while the AP data file has a data object that contains a list of the note IDs, and another that keeps a list of tag IDs and the value associated with that tag:

accessPoints.json:

{
   "accessPoints": [
     {
       "location": {
         "floorPlanId": "b799747a-e2ed-49ad-8c5e-c9ea8c36fa61",
         "coord": {
           "x": 2475.397796817626,
           "y": 1537.8008975928194
         }
       },
       "name": "Simulated AP-1",
       "mine": true,
       "userDefinedPosition": false,
       "noteIds": [
         "37faa8ef-c5c8-4d9d-a882-916db175b935",
         "663419b4-ddb4-4ddb-b3f2-d50233743c5c"
       ],
       "vendor": "Aruba",
       "model": "AP-515",
       "tags": [
         {
           "tagKeyId": "59650f76-3e4b-4c40-b78b-2d0088f5b227",
           "value": "123456789"
         },
         {
           "tagKeyId": "5c9cb127-8ba2-4a60-84e5-75f47ce87f99",
           "value": "C-Suite"
         },
         {
           "tagKeyId": "991b12b7-dbb0-47de-9cd2-260ee064b3e3",
           "value": "aa:bb:cc:dd:ee:ff"
         }
       ],
       "id": "a0b90f2a-8b1b-4339-8362-dc51122931ed",
       "status": "CREATED"
     }
   ]
 }

tagKeys.json:

{
  "tagKeys": [
    {
      "key": "Serial",
      "id": "59650f76-3e4b-4c40-b78b-2d0088f5b227",
      "status": "CREATED"
    },
    {
      "key": "AP Group",
      "id": "5c9cb127-8ba2-4a60-84e5-75f47ce87f99",
      "status": "CREATED"
    },
    {
      "key": "MAC",
      "id": "991b12b7-dbb0-47de-9cd2-260ee064b3e3",
      "status": "CREATED"
    }
  ]
}

notes.json:

{
  "notes": [
    {
      "text": "This is another test note",
      "history": {
        "createdAt": "2020-06-08T16:25:11.868Z",
        "createdBy": "Ian Beyer"
      },
      "imageIds": [],
      "id": "663419b4-ddb4-4ddb-b3f2-d50233743c5c",
      "status": "CREATED"
    },
    {
      "text": "This is a test note",
      "history": {
        "createdAt": "2020-06-08T16:25:04.883Z",
        "createdBy": "Ian Beyer"
      },
      "imageIds": [],
      "id": "37faa8ef-c5c8-4d9d-a882-916db175b935",
      "status": "CREATED"
    }
  ]
}

simulatedRadios.json:

{
  "simulatedRadios": [
    {
      "accessPointId": "a0b90f2a-8b1b-4339-8362-dc51122931ed",
      "accessPointIndex": 2,
      "transmitPower": 0.0,
      "antennaTypeId": "bdf0702a-42be-456a-8891-4cf54940a5c2",
      "antennaDirection": 0.0,
      "antennaTilt": 0.0,
      "antennaHeight": 2.4,
      "antennaMounting": "CEILING",
      "radioTechnology": "BLUETOOTH",
      "spatialStreamCount": 1,
      "shortGuardInterval": false,
      "defaultAntennas": [
        {
          "radioTechnology": "BLUETOOTH",
          "frequencyBand": "TWO",
          "antennaTypeId": "bdf0702a-42be-456a-8891-4cf54940a5c2"
        }
      ],
      "enabled": true,
      "id": "c4f3c521-873c-40de-8076-b1f02b655993",
      "status": "CREATED"
    },
    {
      "accessPointId": "a0b90f2a-8b1b-4339-8362-dc51122931ed",
      "accessPointIndex": 0,
      "transmitPower": 8.000293592441343,
      "channel": [
        1
      ],
      "antennaTypeId": "785280d6-168c-4eab-9819-88e6010e2bef",
      "antennaDirection": 0.0,
      "antennaTilt": 0.0,
      "antennaHeight": 2.4,
      "antennaMounting": "CEILING",
      "technology": "AX",
      "radioTechnology": "IEEE802_11",
      "spatialStreamCount": 2,
      "shortGuardInterval": true,
      "greenfield": false,
      "defaultAntennas": [
        {
          "radioTechnology": "IEEE802_11",
          "frequencyBand": "TWO",
          "antennaTypeId": "785280d6-168c-4eab-9819-88e6010e2bef"
        },
        {
          "radioTechnology": "IEEE802_11",
          "frequencyBand": "FIVE",
          "antennaTypeId": "4ef1637e-06e5-415a-96fd-a97a82273242"
        }
      ],
      "enabled": true,
      "id": "bb7304d1-d564-4018-aa92-e6ca52cba37b",
      "status": "CREATED"
    },
    {
      "accessPointId": "a0b90f2a-8b1b-4339-8362-dc51122931ed",
      "accessPointIndex": 1,
      "transmitPower": 13.979400086720377,
      "channel": [
        36
      ],
      "antennaTypeId": "4ef1637e-06e5-415a-96fd-a97a82273242",
      "antennaDirection": 0.0,
      "antennaTilt": 0.0,
      "antennaHeight": 2.4,
      "antennaMounting": "CEILING",
      "technology": "AX",
      "radioTechnology": "IEEE802_11",
      "spatialStreamCount": 4,
      "shortGuardInterval": true,
      "greenfield": false,
      "defaultAntennas": [
        {
          "radioTechnology": "IEEE802_11",
          "frequencyBand": "TWO",
          "antennaTypeId": "785280d6-168c-4eab-9819-88e6010e2bef"
        },
        {
          "radioTechnology": "IEEE802_11",
          "frequencyBand": "FIVE",
          "antennaTypeId": "4ef1637e-06e5-415a-96fd-a97a82273242"
        }
      ],
      "enabled": true,
      "id": "4ab4a7e1-708d-4f18-b33e-d8891a808e9f",
      "status": "CREATED"
    }
  ]
}

One thing you can do with simulatedRadios.json is go through and adjust your antenna orientations to the nearest 5 or 15 degree increments – having decimal granularity in the antenna orientation isn’t really useful unless you’re doing some very long point to point shots, and it will make the maps look cleaner when your antenna is at 90° instead of 88.6367879° because you manually rotated it by dragging it with the mouse.

I’m also going to omit the antennaTypes.json here, but it’s worth noting that if you have any custom APs/Antennas in your Ekahau setup, that data is included in your project file for portability, and you don’t need that custom config replicated on the next machine that opens up this file, and aren’t limited to the APs and antennas that Ekahau offers by default (although it would be really nice if they made it easy to add custom APs/antennas that survived a code update)

So here’s the basic process to report on your tags and notes:

  1. bring in the list of access points from accessPoints.json. This will get you a list of notes, as well as the tag key IDs, along with that tag’s values.
  2. You’ll need to then cross-reference the tag key IDs from tagKeys.json to get the key values (this approach seems a little convoluted at first, but ensures that a key value can be consistent from one file to the next based on not merely the text in the key value, which will come in to play when merging multiple data files into one. Ekahau was very smart about designing it this way.)
  3. Extract any notes from notes.json.
  4. Cross-reference any additional radio-specific data you need (including orientation) by looking for the access point ID in simulatedRadios.json
  5. Cross-reference any antenna pattern data by looking for the access point ID in antennaTypes.json.
  6. information such as floor number lurks in buildingFloors.json and buildings.json.
  7. and so forth.

Hopefully you’re starting to get the general idea of how this data is put together. It should be a fairly straightforward matter of running a little code against the data file and being able to generate not only a drop list for your installation contractor, but also a big chunk of your configuration script for deploying against your wireless controller. If you’re feeling especially adventurous and saucy, you can even use your wireless system’s API for this and be able to orchestrate a large chunk of your configuration from within Ekahau.

Once I build some actual code, I’ll be sure to share it here.

Here is the big gnarly mind map of the Ekahau data file. It’s probably still very much incomplete and I don’t promise 100% accuracy of data types, but it gives a good visual reference of how the whole thing goes together:

Resolution got smashed by WordPress, so you can check out the full resolution version, or a PDF version.

Working From Home: Home Network

Continuing the series about working from home, today I’m going to talk about the network inside your home, after it gets to your side of the router.

I posted some time ago about solving home wifi woes, so you can read that piece if you’re just trying to fix Wi-Fi weirdness.

In the previous post about internet access, I talked about the router being the gateway between your home network and the rest of the internet. For many home users, your modem, your router, ethernet switch, and your Wi-Fi access point are all stuffed into the same box, which can lead to some confusion when troubleshooting. It also means that if one of those components fails, you likely need to replace the whole thing, which can be a pain. So I’m going to talk about the various components, but just remember that they can sometimes be separate, or sometimes all in that one box we call “router”.

Network Switches

The network switch is the first stop after the router. The switch is what allows you to connect multiple Ethernet devices together. This forms the basis for your entire home network, known as a Local Area Network, or LAN. If you need more ports (not uncommon, since most all-in-one router devices usually only have 4 ports), you can attach a network switch to another. I won’t get into the gory technical details, but this is what allows you to split your network connection among multiple devices. For some homes, 4 ports is enough. For others (such as my own, where I have seven switches comprising nearly a hundred ports), you need to add switches to connect everything.

As a general rule, if a networked device in your house doesn’t move (or is bolted to the structure of the house), you should connect it via a wire, even if it’s capable of wireless. This includes things like TVs, printers, desktop computers, gaming consoles, and so on. A wired network connection will always be more secure and perform better than wireless. If you are a gamer, the reduced latency (“ping”) of a wired connection is something you desperately seek.

Many switches (mostly enterprise grade, but there are growing numbers of small business and home office switches) can also provide DC power over the Ethernet connection – this is known as PoE (and it is spelled out, not pronounced as in “Edgar Allan”), and allows you to power a variety of network devices such as access points and IP phones from a single physical connection. If you have your PoE power source equipment (switch) on a UPS, it can keep all the devices on the network running during a power outage. PoE comes in 3 basic flavors: 15 Watts (802.3af/PoE), 30 Watts (802.3at/PoE+), and most recently, 60 Watts (802.3bt/UPoE). Most devices you’ll encounter at home are perfectly happy to use the 15W variety.

A quick note about network patch cables: Don’t buy into the “Cat 7” marketing hype. This standard doesn’t even exist in the IT world because it doesn’t add any benefit to Ethernet connections. Unless you’re a huge nerd like me, the most you’re ever going to use on your home network is going to be 1 gigabit, which only requires Cat 5e cabling. Buying a more expensive Cat 6, 6a, or 7 cable isn’t going to make your network run any faster (and be very wary of all advice from anyone who tells you otherwise, because they’re about to sell you a whole bunch of crap you don’t need. Cat6 is the norm these days, so it’s probably the cheapest and most common. It will also run 10 gigabit connections within the distances presented in most residential environments. In any case, you’re never going to need 10 gig at home. Not even if you’re a big nerd. See my post about cabling categories for more details.

Wireless

Your Wi-Fi is simply an extension of your home network (LAN) without wires. The device that provides the Wi-Fi signal is called an Access Point, or AP. (Some people still call it a “WAP” for Wireless AP, but that’s not really helpful, because the W could also mean “Wired”). Even inside your residential gateway/router, the access point is a separate piece of hardware that connects internally to the built-in network switch.

The major downside to having an all-in-one gateway device is that what is optimal placement for the gateway (usually where the ISP installer could get a wire through the wall with a minimum amount of effort and damage) is rarely the best place to put an access point. Access points should be centrally located, and the ISP/Cable tech usually likes to be on an outside wall. When you put your wireless there, you’re sending half your signal outside and into your neighbor’s house, especially if you have it turned up to full power to hit the other end of the house.

A recent development in residential Wi-Fi is the rise of “Mesh” devices. This is basically a system of multiple access points which are centrally managed as one system, which allows you to extend wireless throughout your house. “Mesh” refers to those access points themselves connecting to the network wirelessly, rather than using an ethernet connection. Remember what I said earlier about wiring in devices that don’t move? This applies to access points as well. If an access point has to connect wirelessly to your network, it’s going to suffer from all the same wireless problems as any other device. Wire it in unless you have no other option. It’s going to perform a LOT better that way. And, as I mentioned earlier, you may be able to centrally power the access point with PoE.

IoT

As we get more connected, we have more and more smart devices at home. These are collectively referred to as the “Internet of Things”, or IoT. It’s a broad category that includes everything from connected thermostats to smart appliances, wearables such as smart watches, and so on. This is more of a side note to the Work From Home discussion, as IoT is one of those things that potentially impacts a network, but is largely tangential. There’s a saying that “The S in IoT stands for Security”. You’re already saying to yourself, “but there’s no S in IoT!” That’s precisely the point. IoT devices can pose a major security threat on your home network because most of them were not designed with network security in mind. Bottom Line: Isolate them from everything else as much as you can.

Summary/tl;dr

This was just a quick review of your home network components and how they interact, even if they’re all inside the same box. As usual, comments and questions below!

Winnie the Pooh in a honey pot

Working From Home: Firewalls and Honeypots

Yesterday, I saw a social media post from my friend Thorsten, who is an engineer for a large network security company, in which he shared some nifty dashboard graphics from his installation of a nifty little Linux distribution known as T-Pot (I’m a total sucker for great dashboards!).

T-Pot is a collection of various network honeypots with a very nice reporting backend. The project is maintained by Deutsche Telekom, who use it extensively within their own networks. (disclosure: If you run it, it will send back anonymized collected information about the threats seen to their data lake)

So I’m going to veer off a little bit from my regularly scheduled Working From Home series and talk about the importance of securing your networks. T-Pot won’t actually secure your network, it will merely report on the threat actors (most of them automated) that are attacking your network every second of the day. And to a small extent, time they spend “attacking” your honeypot is time they’re not spending attacking real targets (like Pooh up there at the top)

T-Pot takes about 30 minutes to install on a virtual machine (put it in a VLAN that is isolated from everything else!) and then all you do is add a firewall rule to port forward all TCP/UDP (I also did ICMP) to that machine (after any rules to forward to actual stuff), and let it do its thing.

Results will start coming in almost instantly. In a matter of minutes, I’d collected literally hundreds of attacks. After a couple of hours, the numbers were a little disturbing. About 90 minutes after going live, I saw a sharp uptick in one type of attack, as it seems the attackers had found a new target and relayed that information to other attackers.

2.5 hours worth of data.
China, Russia, and.. Canada?
the T-Pot dashboard will show you what usernames and passwords are being used against your honeypot, as well as which known vulnerabilities were being exploited.

If you’re a business hastily trying to get people to work from home, did you just open up a port forward on your Layer 3 firewall to allow Remote Desktop? That probably wasn’t a great idea. As you can see, threat actors are constantly scanning each and every IP address on the internet, probing for vulnerabilities. All it takes is one successful entry into your network, and you’re toast. That can come through your homebound workers as well, if their networks aren’t secure.

Do you still think you don’t need a Layer 7 firewall?