At the beginning of the year, I had all of my SBCs sitting in various boxes, most of them wondering whether they’ll ever see the light of day again but fear not, my various fruity clones, your time is now. One 48-port PoE switch and some IKEA shelving later, we’re in business with the “Ultimate” PoE-Powered SBC Setup.
Ultimately, my goal is to be able to have easy access to every Single Board Computer (SBC) in my collection, allowing me to perform benchmarks at any time, and offer support to projects such as Armbian in the form of hardware for testing of images, etc. That, and I thought it would be a pretty fun project to work on in my new office space!
NOTE: This post contains affiliate links. If you click on these and later make a purchase at that retailer, I may receive a small commission at no cost to you. These types of projects are expensive, even with the extensive support I received so I hope you don’t mind!
Table of Contents
Hardware Breakdown
The Switch
To achieve the PoE paradise I needed hardware. I spent some time scouring used server hardware sites for PoE switches that could do the job and whilst I did find some decent options, I ended up shooting my shot with Ubiquiti as they had the Pro 48 PoE switch that offered PoE+/++ on every port and it’s advertised as having near silent cooling. That’s a big deal when you’re operating in a 12sqm room..
To my delight, they replied quickly and were interested in the project so lucky me, I had a 48-port PoE switch on the way! A 600W power budget is well beyond what I’ll need, even with all 48 PoE ports populated but in theory, that should mean that it’s not working as hard and it’ll be cooler/quieter?
WiFi & Raspberry Pi Zero-style Boards
For the tiny Pi Zero-format SBCs, I have UniFi U7 Pro WiFi Access Point (I purchased this myself!) as I didn’t fancy setting up a bunch of USB Ethernet adapters to handle these to then couple with PoE splitters. Instead, for now, they’ll be WiFi only and powered via a Sabrent 10-port powered USB Hub that can offer up to 60W in total. Plenty for this use case.
I’ve not yet decided exactly how I’m going to “rack” these as I’m keeping an eye out for a cool cluster case, either pre-made or a 3D model to print, so for now, they’re out of picture laying on a table.
In this form factor, we have the Raspberry Pi Zero W/2W, the Radxa Zero, a couple of Orange Pis, the Geniatech RK3566, a couple of Mango Pi, and a few more. I’ll likely look to revisit my Pi Zero Showdown article in the near future!
From The Switch To The SBCs
To get that slightly spicy 52V from the switch to the 5/12V needed on the SBCs, another batch of hardware was required in the form of PoE splitters and PoE HATs. Some SBC vendors are more PoE-friendly than others and either adopt the Raspberry Pi 3/4 layout or at least roll their own with matching PoE HATs.
Sadly Orange Pi and Banana Pi do not partake in this, and as I have a considerable number of their boards, a considerable number of PoE splitters needed to be ordered. Where from? AliExpress of course. I have a mixture of USB-C and micro USB-powered devices but I decided to keep it simple and go with 5V4A USB-C PoE splitters and use USB-C to micro USB adapters where needed. Oh, there were a couple of barrel jack connectors, or some that required 12V so I grabbed a few 12V2.4A DC splitters to either use natively or with a barrel jack to USB-C adapter, like on the Radxa X4L. Time will tell if these will burn my building down.
For the boards that have PoE HATs available, I went down that route because holy moly does that make cabling much, much easier.. For those, I have to say a massive thank you to Libre Computer, and Radxa as they sent almost 30 PoE HATs between them. I’ll be using the Libre Computer PoE HATs on anything that supports the Pi 3/4 PoE layout, and Radxa’s 23W & 25W HATs on their various models as some have a slightly different layout.
Physical Setup
I could have been all professional and attempted this in a proper server rack but there were a couple of obstacles to that approach..
- I’d need rackmount hardware for every SBC and I don’t have a 3D Printer, nor do I have the patience to learn how to design them and get someone else to print them.
- Space and finances were limited
So, I’m in Sweden, and I need affordable furniture so of course, I went to IKEA.
For better or worse, I came across the IKEA Bror shelving unit. It’s made of thin-ish metal and offers some flexibility in adding both shelves and drawers which was perfect for me. Shelves on the top half, drawers at the bottom. Sorted.
I was able to use some fairly affordable plastic clips to make cable management somewhat clean on the sides and underside of shelves and whilst we’ll ignore the spaghetti on the underside of the shelves, I think it came out quite well.
Keeping Everything Cool
Cooling was something I was aware of as whilst I’m building this in the middle of Swedish winter and the sun hasn’t been seen for months, at some point, it’s going to return and for that short week or two, I have black metal shelving in front of 2 large windows. Sure, I could just close the blinds but I like the sun sometimes so that’s not feasible.
Instead, I bought a massive amount of 20mm high nylon standoffs and nuts (M2.5 size) to raise everything off of the metal shelves, and I’ve placed some 120mm fans at the end of each shelf to get some air moving. Some of the PoE HATs and SBCs have fans, and I may keep those depending on the noise levels, however, I do plan to add some simple heatsinks to any bare SoCs to help too.
Network and Power
I want to pretend that I’ve put in a load of effort here but honestly, it’s fairly straightforward. I use the UniFi Network defaults extensively, leaving it to its own thing and I simply have static IPs set up for each device via DHCP. At most, I’ve separated all of the SBCs off onto a somewhat isolated network, away from my other devices and other servers. This is 50% for “security” reasons and 50% for “Oh I know if it’s in X range it’s an SBC”.
For power, it’s much the same. The USW Pro 48 PoE switch has 40 PoE+ ports and 8 PoE++ ports. Very few of my devices require PoE++, but it’s nice to have and a couple of things are there (an N100 mini PC, and the Radxa X4 so far).
I hit a snag, however, as I ran out of ports on the USW Pro 48 PoE switch and needed more than the 8-port 2.5GbE switch could provide. I went searching on Amazon so I could get something quick and ended up with this 16-port 1GbE PoE switch. It seems fine so far. It’s fanless, unmanaged, and claims to offer 300W but I have no intention of pushing it that far to find out. I hope to replace it with another Ubiquiti soon.
For some other devices that can take 2.5GbE, I have an 8-port 2.5GbE PoE+ switch connected to the USW-Pro-48-POE via 10G SFP+. I’ll likely have an iperf3 endpoint connected via 10G because why not.
The 600W power budget of the UniFi switch means around 12.5W per port if they were all utilised equally at the same time but honestly, 99.9% of these devices are going to be idle most of the time, drawing between 1-4W. When the time to benchmark something comes, it will be done in a controlled manner, with only a few boards at a time. Or if I want to make sure everything is done “properly” it’ll be moved to the workbench and the actual benchmarking station (more on that in another article soon!)
Setting up the Single Board Computers
Software
This post is already getting long enough, so I’ll cover this in more depth in a separate piece but with so many SBCs, the operating systems and how they’re managed becomes a rather big undertaking.
I ummed and ahhed about how to go about this for a while. Do I use the vendor images, regardless of their current state? Do I use a mix of somewhat up-to-date vendor images and something like Armbian for the others? In the end, I decided to do neither of those and went with Debian-based Armbian (without desktop environments) on everything with working images.
I chose Armbian because they offer an extremely wide range of images for boards across my collection, and they have great tooling to build images to my needs, and armbian-config along with the automation for configuring new boards makes it quick and easy to get running.
This means I’ve ended up with images with a range of support statuses, however when dealing with this number of devices, having a somewhat uniform software setup makes things much, much easier. I’m very tempted to name and shame those who have abandoned their boards in the follow-up piece but let’s see how merciful I’m feeling on the day.
Storage
Whilst a lot of the older boards only support microSD cards, a lot also have eMMC and/or M.2 NVMe connections available and I’m trying to remove the need to go and plug things in on-demand with this setup.
Every board is getting a 32GB Raspberry Pi microSD Card (at least those with microSD card slots). 64GB would have offered more performance but honestly, given most of these devices will have eMMC and NVMe, the added cost wouldn’t have been worth it.
Luckily, over the years I’ve amassed a range of eMMC modules for the different boards and will be able to populate most of the more powerful boards with some kind of eMMC. Those with M.2 NVMe options will get either an Intenso 250GB or Silicon Power 256GB drive and whilst they’re not the speediest of drives, they’re going to max out the performance of most SBCs they’re put in. I’ll be grabbing a few faster drives to put in those that offer a few more PCIe lanes at some point but it’s not a priority.
How Are Things Looking?
Now that everything is installed and powered on, I thought I’d touch on the power and cooling situation with everything in place.
With all RJ45 ports populated and powered up on the UniFi switch, we’re looking at around 160-165W which gives us plenty of headroom. Before we get onto the temperatures of the boards themselves, let’s have a quick look at the UniFi switch. Drawing the 166.8W shown above, the switch’s fan is running at a reasonable RPM and the temperature reads 50 degrees Celsius (in a room that’s currently around 22c ambient.)
Ubiquiti advertises the switch as having “near-silent” cooling (you can see the thin slots above the ports in the photo above, they’re the intake grilles) and whilst it’s definitely nowhere near Pratt & Whitney PW4170 (one for the Aviation people out there) levels, it’s audible. I don’t think I can blame Ubiquiti for that though, I’m probably not supposed to run it in a 12sqm room like this.
Considering the setup as a whole, we’re close to 270W of power draw across the 3 switches, router, and access point which isn’t completely terrible.
Challenges and Solutions
Well, at this point, this section could be a separate piece, but I’ll try to condense it down. At first, I spent an embarrassingly long time trying to figure out how to actually rack things up. With the boards that had PoE HATs, it felt easy. Slap some standoffs on and be done with it.
For PoE splitters, however, I wanted to try and think of something a little more elegant. I’d always wanted to try DIN rails out and thought they could be perfect but alas, they were not. All of the boards having their RJ45 and power connections on different parts of the board threw multiple ideas too.
Whilst they looked nice, mounting them on my shelving was a pain in the rear end and accessibility was going to suck. I want to take a moment to shout out KKSB Cases who sent over the above DIN rail mounts for the Raspberry Pi for me to check out. They’re great, but the solution as a whole wasn’t for me, so I went back to the drawing board.
Most of the issues I faced would have been avoided if I’d spent a bit more time planning and double-checking realistically as the next problem was whilst I had PoE HATs, the wide range of boards I had to put them on had a range of different components on them, meaning I needed GPIO/PoE header extensions. I bought the 1st ones on Mouser and incredibly, the 40-pin GPIO and 4-pin PoE headers were different heights.
That meant I had to go hunting on AliExpress and I ended up ordering 30 4-pin PoE header extensions, 40-pin, and 6-pin GPIO extensions just to be safe. It was overkill, but I’m now prepared for the worst, and these worked perfectly.
Going back to the PoE splitter setup, I’ve decided to strip it back and use any cases I already had for the various boards and stick the splitters to those using small command strips. If I don’t have a case yet, then well, I got creative and stuck them to the USB/RJ45 ports on the boards themselves. Elegant? No. Functional? Kind of.
Final Results and Future Plans
If you’ve made it this far, I salute and thank you, it’s not the style of article I’m used to writing and it threw me a little. Though saying that, it’s felt good.
If I were to try something like this again though, I’d definitely try and do a little more research, or obtain a few more useful skills to be able to undertake things myself so I don’t have to rely on off-the-shelf products so much, especially niche/hard to find ones!
I’m going to to reveal a little more about the plans for the setup in due course, however for now, it’s been a fun data-gathering project and will allow me to get quick access to any of the boards I have at any time. This will be especially useful for helping people out with the testing of OS images, and software on a wide range of ARM/RISC-V systems.
Conclusion – The End?
I’ll be revising this over time and want to try and make it a little more elegant but for now, I think we’re in a good place to call version 1 “finished”. I want to look into getting a 3D printer to try and put together a more uniform setup but I just don’t have the time right now to dive into research, especially after the Bambu unit I was looking at before is out of the window. This is why I can’t have nice things.
This has been a great first project for the new office space, and hopefully, you’re going to love what I’ve got coming in the coming month or so. If you don’t, well, it’s going to suck so please like it. Please.
To sign off, I’d like to once again give a massive thank you to everyone who has contributed to the project thus far, it’s only the beginning, and over time, I hope for this to grow into a useful resource for open-source/community projects. Once I’ve done my initial testing, it will sit largely idle and I don’t want this to go to waste.
So thank you to Ubiquiti, Radxa, Libre Computer, HackerGadgets, Debix, and everyone I’ve bugged in DMs for ideas/feedback, I hope it’s all (going to be) worth it!
