Hey! I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it’s working great, but I want to learn MOAR and I need help…
Recently, I’ve been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don’t understand how things are supposed to be done. All the examples I find feel wrong. More specifically:
- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f
?? It feels wrong and way too “by hand”! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it’s really not.
It’s very much a noob question, but I really want to understand what I am doing wrong. I’m really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
I’ll post more later (reply here to remind me), but I have your exact setup. It’s a great way to learn k8s and yes, it’s going to be an uphill battle for learning - but the payoff is worth it. Both for your professional career and your homelab. It’s the big leagues.
For your questions, no to all of them. Once you learn some of it the rest kinda falls together.
I’m going into a meeting, but I’ll post here with how I do it later. In the mean time, pick one and only one container you want to get started with. Stateless is easier to start with compared to something that needs volumes. Piece by piece brick by brick you will add more to your knowledge and understanding. Don’t try to take it all on day one. First just get a container running. Then access via a port and http. Then proxy. Then certs. Piece by piece, brick by brick. Take small victories, if you try to say “tomorrow everything will be on k8s” you’re setting yourself up for anger and frustration.
@sunoc@sh.itjust.works Edit: To help out I would do these things in these steps, note that steps are not equal in length, and they are not complete - but rather to help you get started without burning out on your journey. I recommend just taking each one, and when you get it working rather than jumping to the next one, instead taking a break, having a drink, and celebrating that you got it up and running.
Start documenting everything you do. The great thing about kubernetes is that you can restart from scratch if you have written everything down. I would start a new git repository with a README that contains every command you ran, what it did, and why you did it. Assume that you will be tearing down your cluster and rebuilding it - in fact I would even recommend that. Treat this first cluster as your testing grounds, and then you won’t feel crappy spinning up temporary resources. Then, you can rebuild it and know that you did a great job - and you’ll feel confident in rebuilding in case of hardware failure.
Get the sample nginx pod up and running with a service and deployment. Simply so you can
curl
the IP of your main node and port, and see the response. This I assume you have played with already.Point DNS to your main node, get the nginx pod with
http://your.dns.tld:PORT
. This should be the same as anything you’ve done with docker before.Convert the yaml to a helm chart as other have said, but don’t worry about “templating” yet, get comfortable with
helm install
,helm upgrade -i
, andhelm uninstall
. Understand what each one does and how they operate. Then go back and template, upgrade-ing after each change to understand how it works. It’s pretty standard to template the image and tag for example so it’s easy to upgrade them. There’s a million examples online, but don’t go overboard, just do the basics. My (template values.yaml) usually looks like:<<servicename>> name: <<servicename>> image: repository: path/to/image tag: v1.1.1 network: port: 8888
Just keep it simple for now.
istio
. I can go into more details why later, but I like that I can create a “VirtualService” for "$appname.my.custom.tld` and it will point to it.nginx.your.tld
and be able to curlhttp://nginx.your.tld
and see that it routes properly to your sample nginx service. Congrats, this is a huge one.Certificate
types in k8s. You’ll need to use the proxy in the previous step to route the /.well-known endpoints on the http port from the open web to cert-manager, for Istio this was another virtual service on the gateway - I assume Traefic would have something similar to “route all traffic on port 80 that starts with /.well-known to this service”. Then, in your nginx helm chart, add in a Certificate type for your nginx endpoint,nginx.your.tld
, and wait for it to be successfully granted. With Istio, this is all I need now to finally curlhttps://nginx.your.tld
!At this point you have routing, ports, and https set up. Have 2 drinks after this one. You can officially deploy any stateless service at this point.
Now, the big one, stateful. Longhorn is a bear, there are a thousand caveats to it.
Step one is where are your backups going to go. This can be a simple NFS/SMB share on a local server, it can be an s3 endpoint, but seriously this is step 1. Backups are critical with longhorn. You will fuck up Longhorn - multiple times. Losing these backups means losing all configs to all of your pods, so step one is to decide on your stable backup location.
Now, read the Longhorn install guide: https://longhorn.io/docs/1.9.0/deploy/install/. Do not skip reading the install guide. There are incredibly important things in there that I regretted glossing over that would have saved me. (Like setting up backups first).
The way I use longhorn is to create a PV in longhorn, and then the PVC (you can look up what both of these are later). Then I use Helm to set what the PVC name is to attach it to my pod. Try and do this with another sample pod. You are still not ready to move production things over yet, so just attach it to nginx.
exec
into it, write some data into the pvc. Helm uninstall. See what happens in longhorn. Helm install. Does your PVC reattach? Exec in, is your data still there? Learn how it works. I fully expect you to ping me with questions at this point, don’t worry, I’ll be here.Longhorn will take time in learning, give yourself grace. Also after you feel comfortable with it, you’ll need to start moving data from your old docker setup to Longhorn, and that too will be a process. You’ll get there though. Just start with some of your lower priority projects, and migrate them one by one.
After all of this, there is still more. You can automount smb/nfs shares directly into pods for media or anything. You can pass in GPUs - or I even pass in some USB devices. You can encrypt your longhorn things, you can manage secrets with your favorite secret manager. There’s thousands of things you’ll be able to do. I wish you luck, and feel free to ping me here or on Matrix (@scrubbles@halflings.chat) if you ever need an ear. Good luck!
Great writeup.
Very true. Each brick you lay upgrades your setup and your skillset. There are very few mistakes in Kubernetes as long as you make sure your state is backed up.