18 Mar 2019 by Lior Kamrat
So I talked about the motivation for this project but I think what you really care about is what kind of hardware did I choose.
Warning: this is going to be a very long post. Usually, when I am doing multi-part series I am breaking the posts as much as I can but this time, I wanted you to have all the hardware related details in one page. Hope you will not give up 😉
First thing first, let me remind you of the key requirements and guidelines I worked with:
I worked under the assumption that a single K8s master node is a no-go.
This had to be as plug-and-play as possible, meaning, I wanted easy access for how power gets connected.
If one Pi gets malfunctioned, I needed a way to easily replace it.
The network stack had to be compact but also managed by software. I wanted to have granular control for some more advanced use cases that might come in the future.
I didn’t want to deal with the “not enough power” situation.
I like things to stay cool.
It had to be pretty as much as possible. Functionality and portability come first but if I can make it look nice why not?!
There is already a V2 for this setup that I am starting to wrap my hand around but I will get to that later.
Choosing the Pi’s was the easiest part of this hardware selection process. I went with the latest and greatest Raspberry Pi 3 Model B+.
All of the blog references I presented in the previous post are all talking about a single master cluster. This is for sure a great start and I was able to have this setup up and running fairly easily after multiple tries but I wanted to take it one step further and have a multi-master Kubernetes cluster (so 3 masters nodes) for few reasons:
Learning a bit more about all the gouaches of kubeadm.
Work with a more complex automation environment I will have to design.
To learn and to challenge myself.
Because the 3 masters need a load-balancer, another Pi was designated to be the HAProxy, plus, two worker nodes which bring us into a total of 6 shiny Raspberry Pi’s.
This is where things got interesting. If you look at the list of requirements some of you might think “something gotta give” which is exactly what I said.
After scouting the internet, talking to people and reading around 20 something reviews, I decided to go with the 8 Slot Cloudlet Cluster from C4LABS.
This case has some very nice features which made me believe that I might not have to give up on any of my requirements.
You get everything you need. The instructions provided were good and all you need is just a basic screwdriver set.
Each Pi gets its own acrylic plate. Those plates are designed for easy and safe removal of each Pi individually from the so you won’t have to break everything if you need to take one out.
I will say that once everything is connected it can be a bit harder to remove a single Pi but this is because I really utilized the space of the case so it made things tight.
This case is very solid and has a great build-out quality. The holes in the back provided me with a convenient way to plug the cluster without having to go through loops. More details on the extra research needed here later…
It comes with 4, 2-pin, 50x50mm fans which will provide the right amount of cooling. Connecting those fans was pretty easy and straightforward. I will say though that they are not super quiet but IMO it’s manageable.
I found this case providing a great balance between functionality and portability. It can fit a lot as you will see in the pictures at the end of this post but it’s also lightweight and small enough to get in your bag or carry-on which is exactly what I wanted.
For those who are not familiar with Raspberry Pi, it uses Micro USB. Going with individual power supply for each Pi was not an option since this will kill the portability aspect of things (plus, it’s ugly) so I had to search for the right USB hub.
Choosing the right USB HUB was not an easy challenge:
It had to have at least 6 ports for the 6 Pi’s.
Produce enough juice.
The hub power adapter jack had to be accessible from holes at the back of the case – I know right?!
Need to be small, very small.
Eventually, after not the easiest research, I decided that the Anker USB 3.0 7-Port Hub was the right choice. Few reasons why:
The power adapter jack seems to be right in the right place. I had no real way to know it fit perfectly but I’m happy I got it right 🙂
7 ports producing enough power to 6 Pi’s and change.
Small and elegant. When I looked at its dimension (4.29 x 1.73 x 0.91 inches), it seemed like the perfect fit. I bought the case before the hub so I was able to measure the exact depth of the case.
Like I said, the power-related research was a bit tedious. Few links you might find useful here:
For the selection of the Micro USB cables, I had few guidelines:
I didn’t want some generic, low-quality cable which will pop on an initial search.
It needs to be able to go from the USB hub to farthest Pi in the case with being able to extract the Pi easily if needed.
It needs to be able to go from the USB hub to it’s closest Pi in the case without leaving lot’s of cables to hang around.
Once all the cables are in and connected, the second hold (on the left in frontal view) had to be kept clear for the network switch power jack to come.
The 6 cables I needed had to preserve the airflow in the cases optimal as possible.
Any color other than boring black or white would be a bonus.
So you can see, even though “it’s just a cable”, with this kind of setup everything needs to fit. Eventually, for the USB cables, I selected Anker again and went with the red PowerLine 1ft Micro USB.
As you can see from the picture, pretty slick!
A pro tip: When connecting the cables, you want to start from the far left Pi so it will be easy to organize the cables and click-in the Pi’s.
Surprisingly enough and unlike most of the projects I was involved in, for this project, the network design was easier then I thought but I did have some challenges with the biggest one being not able to find an 8-port router/switch to meet my needs.
The case is designed to fit a network switch beneath the Pi’s. My guidelines here were:
6 Pi’s + external Router port + laptop = 8 port switch
1GB support (dah…)
Managed switch, nothing less.
Compact enough to fit right in without going way over the case edges.
Power jack location which will be accessible from the hole at the back of the case.
Once again, I waited for the case to arrive and then started my research around dimensions. Luckily, my first choice was also the last one. I am a Ubiquiti fan so I decided to go with the UniFi Switch 8 (US-8).
What can I say other than great product, super easy to manage, has more capabilities than most managed-switches at its level, fits right in and stylish!
The CAT6 cables to connect the Pi’s to the switch had to be short unless the entire would have looked awful and messy. Fortunately, I was able to find the SlimRun Cat6 Glow in the Dark Ethernet Patch Cables from Monoprice which comes in multiple lengths so going with 0.5ft (15.24cm) seems like the obvious choice.
I thought the “glow in the dark” will be a nice addition but as for now. I don’t see any “glowing” :-). I don’t really care so oh well…
For internet connectivity, I had to give up the idea of trying to find a solution that will be embedded inside the case. Nevertheless, I still wanted a router that will be compact but that will support two of those setup’s (two cases with 6 Pi’s each).
Staying on the Ubiquiti track, I decided to go with UniFi Security Gateway (USG).
Small, portable and pretty.
Complete the management experience alongside the UniFi Switch.
Has two LAN ports which can connect to two cases.
Pi has Micro SD port for loading your operating system and storing data. For each Pi I got the MicroSDHC EVO Select 32GB Memory Card. It was a very much straight forward choice.
For the current version of the project, I didn’t plan for persistent storage the Kubernetes pods will leverage.
The SD card cannot be used for that, it’s just not the right storage solution for the random reads/writes, the durability and reliability a Kubernetes environment needs. I put some notes in the next section around what I am planning on that space.
Throughout the process, I collected some learning and took some notes which I planned to use in the next version of this project that I would like to share with you.
Need to have quieter fans as the ones coming with the case are too loud IMO. It’s not horrible but if you are demoing this thing in a meeting room it can be annoying.
In addition to that, it will help the fan cables to be just a bit longer. That way, I will be able to extract a single Pi without being afraid the cable will snap from the fan or twist the pins its connected to on the Pi.
Persistent storage will require some physical design changes. I am looking to connect Geekworm Raspberry Pi X850 mSATA SSD Extension and Samsung 860 EVO 250GB mSATA Internal SSD drive to the right Pi and use it as an NFS mount point (on top of it being the HAProxy).
Will have to do some tests around power and see how everything fits inside the case. I am optimistic about it but will know for sure later on…
This means persistent storage SPoF. Will have to get creative here.
Giving 2-3mm bump to both the switch and USB hub can help to align it to its power jack location with the holes at the back of the case. I’m not super worried about it as it fits nicely as it is. Yes, it was a VERY long post indeed but I wanted you to have the entire process in a single page.
In the next posts, I will start going over the software and the Kubernetes side of things. There still a few issues I am facing when it comes to K8s, etcd and ARM but I am furiously working on those.
Comments are closed for this post.