<![CDATA[Kubernetes (and k3s) is a damn headache.]]>Kubernetes (and k3s) is a damn headache.

So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.

Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.

Do y'all have suggestions for FLOSS k3s alternatives?

For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3s

Nice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)

Your suggestions are very welcome

(Meme nabbed from https://lemmy.world/post/13302376)

]]>https://forum.fedi.dk/topic/f4f5f803-45d4-4e81-8e25-36373c579e85/kubernetes-and-k3s-is-a-damn-headache.RSS for NodeThu, 03 Sep 2026 04:19:06 GMTWed, 26 Aug 2026 10:49:40 GMT60<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Thu, 27 Aug 2026 17:14:37 GMT]]>@papiris I got a bit fed up with ansible today. Have you heard of mgmt? https://mgmtconfig.com/

I only looked at the documentation and a few examples. It's "let's mutate the machine" like ansible, but you have a proper programming language for it with proper dependency graphs. Seems much more sane to me. I might experiment with it and consider moving data.coop over to it.

]]>https://forum.fedi.dk/post/https://social.data.coop/users/reynir/statuses/117168595136002492https://forum.fedi.dk/post/https://social.data.coop/users/reynir/statuses/117168595136002492Thu, 27 Aug 2026 17:14:37 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Thu, 27 Aug 2026 15:30:47 GMT]]>@happyborg
imv, proxmox overreaches by taking control of the OS layer. no, thank you, i want to run and completely control that part (debian stable with my own admin processes). is a better, more flexible, simpler option for the virt/container layer.
@shivjm @papiris

]]>https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117168186849871944https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117168186849871944Thu, 27 Aug 2026 15:30:47 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Thu, 27 Aug 2026 15:26:21 GMT]]>@monotux
regarding storage: ceph is again, quite complex. alternatively, you can get pretty far with zfs snapshots and replication (https://klarasystems.com/articles/zfs-vs-ceph-do-you-actually-need-ceph/).
i'll also be deploying garage soon for s3 compat distributed objstore, which is built to be run by ppl without datacenters (https://garagehq.deuxfleurs.fr/)
@papiris

]]>https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117168169367440533https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117168169367440533Thu, 27 Aug 2026 15:26:21 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 17:43:19 GMT]]>@papiris what you are asking for is....kubernetes. offload state to some other machine if possible (db, storage...) and use some dedicated OS like Talos and reduce the amount of maintenance. argo or flux for gitops, for talos talhelper is very nice and declarative.

if you can live with giving up some features, have a look at incus or proxmox. but if you want storage with high availability you are probably going to use ceph and the complexity increases.

]]>https://forum.fedi.dk/post/https://hachyderm.io/users/monotux/statuses/117163045638537776https://forum.fedi.dk/post/https://hachyderm.io/users/monotux/statuses/117163045638537776Wed, 26 Aug 2026 17:43:19 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 14:30:04 GMT]]>@papiris
in my view, we need smaller components for a composable, simpler, understandable stack. my choices:
stable+backports
for storage
for container and virt mgmt
(with debops) for cfgmgnt

]]>https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117162285770286078https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117162285770286078Wed, 26 Aug 2026 14:30:04 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 14:27:52 GMT]]>@papiris
completely agree, that's tech for corporations, not human scale, it is de facto closed off to us and centralized by way of complexity. it can never be our tech.
your requirement list looks to me still very influenced by corp scale instead of our real needs. do you *really* *need* all that? that is for you to decide.
1/2

]]>https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117162277104000859https://forum.fedi.dk/post/https://mstdn.io/users/zeh/statuses/117162277104000859Wed, 26 Aug 2026 14:27:52 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 13:15:13 GMT]]>@shivjm I'm new to this but second Proxmox. It's been remarkably easy to get going with and seems very capable.
@papiris

]]>https://forum.fedi.dk/post/https://fosstodon.org/users/happyborg/statuses/117161991461639429https://forum.fedi.dk/post/https://fosstodon.org/users/happyborg/statuses/117161991461639429Wed, 26 Aug 2026 13:15:13 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 11:50:10 GMT]]>@papiris i think docker swarm can be this. Although not sure if this is a still alive project.

]]>https://forum.fedi.dk/post/https://mastodon.social/users/simon_schwarz/statuses/117161656996950550https://forum.fedi.dk/post/https://mastodon.social/users/simon_schwarz/statuses/117161656996950550Wed, 26 Aug 2026 11:50:10 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 11:05:33 GMT]]>@abortretryfail yup

Seems like most tools are either made with large IT-orgs in mind, or assume reliable underlying infra

]]>https://forum.fedi.dk/post/https://hachyderm.io/users/papiris/statuses/117161481593097529https://forum.fedi.dk/post/https://hachyderm.io/users/papiris/statuses/117161481593097529Wed, 26 Aug 2026 11:05:33 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 11:04:48 GMT]]>@papiris If you’re open to leaving Kubernetes, (which I use for my own stuff) does a lot of this but not all. I think the major thing is that it doesn’t inherently interact with Git in any way, so it’s up to you to implement GitOps. It might be worth looking at:

https://proxmox.com/en/

I *much* prefer it to any form of Kubernetes.

]]>https://forum.fedi.dk/post/https://mastodon.social/ap/users/116306625129462789/statuses/117161478629587957https://forum.fedi.dk/post/https://mastodon.social/ap/users/116306625129462789/statuses/117161478629587957Wed, 26 Aug 2026 11:04:48 GMT<![CDATA[Reply to Kubernetes (and k3s) is a damn headache. on Wed, 26 Aug 2026 10:58:17 GMT]]>@papiris
5 years ago I might've suggested Hashicorp's Nomad.

That's a tall order of requirements for sometimes simple and low maintenance. 🙁

]]>https://forum.fedi.dk/post/https://mastodon.social/users/abortretryfail/statuses/117161452970627747https://forum.fedi.dk/post/https://mastodon.social/users/abortretryfail/statuses/117161452970627747Wed, 26 Aug 2026 10:58:17 GMT