Kubernetes on Leftovers

About 1703 wordsAbout 6 min

2025-09-01

Repurpose Old Hardware

I had plans to use some spare time during the summer to write a heavy thought piece on what can be done on individual, corporate, and government levels to tackle some of the global risks today by working for something rather than against something, which seems to be the general stance these days (I am against the impacts of climate change as compared to being for a carbon-neutral fossil-free future). Instead I did something much more fun. Repurposed some old computers. At least that is what I started out doing..

I have some computers of old collecting dust that I have been meaning to setup as a Kubernetes lab environment. Another blog post might explain why I think, going forward in IT and tech, having access to platform agnostic containarized hosting is going to be even more important than it already is. Let´s focus on the technical aspects of getting some use out of old hardware in this post.

Objects intended to be used for this experiment:

HP ProDesk something-something 8GB RAM Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz CPU 128GB SSD - This used to be a SMB Server with a Windows Server 2019 license attached to it.
Sony Vaio VPC laptop 6GB RAM 128GB SSD - This used to be my main driver working MOSS 2007 (SharePoint) projects back in the day.
HP ProLiant something-something 1GB RAM Intel Pentium CPU 512GB HDD - This used to act as a network file server running Ubuntu
Netgear WGT624 router - 20 years ago this was the king in home networking. The solid 4-port 100 Mbps ethernet switch is the only part we are interested in on this.

The laptop I quickly realized have heating issues and shutdown when under load. The HP Proliant used to have 2GB of RAM but after a power outage it is down to 1 and the HDD is in a bad state. Not going to be much of a cluster with only one computer and I do not have the time to do hardware repair at the moment. What else can be used?

Smartphones! You are perhaps the better citizen then I am and use different return programs to get the materials recycled in your obsolete electronics. As you have likely figured out, I do not do that. After some internet searching, I concluded that from the box storing my old phones the Xiaomi Mi A1 was a decent candidate for this because others have successfully flashed and installed Linux on them.

Some say that a modern smartphone has more compute power than existed on the planet in 1980. The Xiaomi Mi A1 was a midrange phone when released in 2017 and considering the exponential growth in compute power I say it has at least the compute power equivalent to what existed worldwide in 1977 (Chat-GPT confirms it has more, so it must be true). Incidentely 1977 is the magic year when the first Star Wars movie was released, and since I will use 2 of these (I get my wife the same type of phone when I get a new..) there will be 2x the compute power that existed on the planet when Star Wars was released added to the cluster.

Objects added to experiment:

2 x Xiaomi Mi A1 - 4GB RAM 2GHz octa-core Snapdragon 625 with 650 MHz Adreno GPU and 64GB storage

We are now levelling up this experiment. Originally it was going to be a cluster of all x86 Intel-based devices running latest LTS of Ubuntu Server connected by ethernet. Now we are going for a hybrid Kubernetes cluster mixing ARM and x86 architecture devices connected by both Wi-Fi and ethernet and running different Linux distributions. I will not be surprised if this gives me headaches later on, but for now let´s consider it added fun, and I must admit I am a bit excited to see if I can get some small LLMs running on those old midrange Snapdragon devices.

OS Install

There are different distributions of Kubernetes. The original is called K8s and is suitable for large scale production deployments and has some minimum requirements that we cannot fulfill in this setup. We are going for K3s which are more suitable for IoT, Edge Computing and Home Labs, with lower hardware requirements. Arguably I should be using MikroK8s for this, but I am not finding many success stories on doing so on low powered ARM devices such as the old Xiaomi phones I am using. What the distributions share is however that they run as a Linux service and thus we need a Linux distribution on all intended nodes in the cluster.

Installing Ubuntu Server 24.04.3 LTS on the HP ProDesk was a breeze. Nothing much to write about it. I used bootable USB for installation media and connected it by ethernet to the Netgear WGT624 that I had already disabled radio interface and DHCP on, so that it effectively operates as a dumb switch. I then reserved the IP the device had been assigned from my home network router.

For getting Linux and K3s on the smartphones I was inspired by this great blog post that explains how to get PostmarketOS and K3s running on a OnePlus 5T and a OnePlus 8 Pro. PostmarketOS is a Linux distribution suitable for mobile devices that is based on Alpine Linux. The PostmarketOS wiki has a page dedicated to the Xiaomi Mi A1 that I could largely follow with success.

Some notes regarding getting PostmarketOS installed that might save you some time, unless you want to go deep in the documentation:

Install and run pmbootstrap on a Linux client. I wasted time trying to get it running using WSL.
If you do a minimal build with no GUI, as I did, then WiFi will not be enabled automatically. After successfull flash and boot into PostmarketOS, keep the device USB connected to the Linux client and use the USB ethernet functionality to do initial SSH (SSH yourusername@172.16.42.1). Enable WiFi (nmcli radio wifi on) and then connect to your network (nmcli d wifi connect yournetworkSSID --ask).

K3s

Since I was inspired by previously mentioned blog post I knew that the kernel config for PostmarketOS might not have all features enabled that is required for running K3s. Luckily there is a config checker for that.

Download it:

wget -O check-config.sh https://raw.githubusercontent.com/k3s-io/k3s/refs/heads/master/contrib/util/check-config.sh

Run it:

sudo ./check-config.sh

My output:

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroups Nonexistent (fail) 
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_IP_NF_TARGET_REJECT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_STATISTIC: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: missing
      - CONFIG_XFRM_USER: missing
      - CONFIG_XFRM_ALGO: missing
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 1 (fail)

Alright, so the config checker identifies an issue that is "Generally Necessary" and some missing "Optional Features". The optional features I do not really care about at this stage, but the generally necessary might be blocking K3s from installing so let´s take a look at that one.

A few internet searches later and I believe this might be a false positive. Depending on how I check for cgroup version I get different results.

~$ stat -fc %T /sys/fs/cgroup/
UNKNOWN

~$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

I interpret this as the kernel actually do have a cgroup version compatible with K3s, so let´s go ahead and install, keeping this in mind in case it fails and we need to recompile the kernel and flash devices again.

Next, install K3s!

On the Ubuntu Server:

curl -sfL https://get.k3s.io | sh -

The above will install K3s service as a server node with defaults, which includes kubectl, crictl and ctr.

Next run the following to retrieve the token from the server node that is needed to join agents to it.

sudo cat /var/lib/rancher/k3s/server/node-token

On the PostmarketOS devices:

curl -sfL https://get.k3s.io | K3S_URL=https://myserverip:6443 K3S_TOKEN=mynodetoken sh -

Replace myserverip with the IP of your server node and mynodetoken with the output from previous command.

What we have now installed is a basic cluster of:

To confirm this you can view what nodes are in the cluster by running:

kubectl get nodes

That is it. We now have Kubernetes running on some old, otherwise dust collecting, devices. Actually getting some use from the setup involves additional steps like installing applications that are generally needed, like a Certificate Authority, a Load Balancer, a DB Server, and a working CI/CD setup for my own development.