Kubernetes (and k3s) is a damn headache.
-
Kubernetes (and k3s) is a damn headache.
So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.Do y'all have suggestions for FLOSS k3s alternatives?
For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3sNice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)Your suggestions are very welcome
(Meme nabbed from https://lemmy.world/post/13302376)
#fedihelp #kubernetes #k3s #coopcloud #incus #homelab #IT #techcoop #riseAgainstBigTech #reticulum
@papiris If you’re open to leaving Kubernetes, #Proxmox (which I use for my own stuff) does a lot of this but not all. I think the major thing is that it doesn’t inherently interact with Git in any way, so it’s up to you to implement GitOps. It might be worth looking at:
I *much* prefer it to any form of Kubernetes.
-
@papiris
5 years ago I might've suggested Hashicorp's Nomad.That's a tall order of requirements for sometimes simple and low maintenance.
@abortretryfail yup
Seems like most tools are either made with large IT-orgs in mind, or assume reliable underlying infra
-
Kubernetes (and k3s) is a damn headache.
So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.Do y'all have suggestions for FLOSS k3s alternatives?
For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3sNice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)Your suggestions are very welcome
(Meme nabbed from https://lemmy.world/post/13302376)
#fedihelp #kubernetes #k3s #coopcloud #incus #homelab #IT #techcoop #riseAgainstBigTech #reticulum
@papiris i think docker swarm can be this. Although not sure if this is a still alive project.
-
@papiris If you’re open to leaving Kubernetes, #Proxmox (which I use for my own stuff) does a lot of this but not all. I think the major thing is that it doesn’t inherently interact with Git in any way, so it’s up to you to implement GitOps. It might be worth looking at:
I *much* prefer it to any form of Kubernetes.
-
Kubernetes (and k3s) is a damn headache.
So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.Do y'all have suggestions for FLOSS k3s alternatives?
For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3sNice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)Your suggestions are very welcome
(Meme nabbed from https://lemmy.world/post/13302376)
#fedihelp #kubernetes #k3s #coopcloud #incus #homelab #IT #techcoop #riseAgainstBigTech #reticulum
@papiris
completely agree, that's tech for corporations, not human scale, it is de facto closed off to us and centralized by way of complexity. it can never be our tech.
your requirement list looks to me still very influenced by corp scale instead of our real needs. do you *really* *need* all that? that is for you to decide.
1/2 -
@papiris
completely agree, that's tech for corporations, not human scale, it is de facto closed off to us and centralized by way of complexity. it can never be our tech.
your requirement list looks to me still very influenced by corp scale instead of our real needs. do you *really* *need* all that? that is for you to decide.
1/2 -
Kubernetes (and k3s) is a damn headache.
So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.Do y'all have suggestions for FLOSS k3s alternatives?
For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3sNice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)Your suggestions are very welcome
(Meme nabbed from https://lemmy.world/post/13302376)
#fedihelp #kubernetes #k3s #coopcloud #incus #homelab #IT #techcoop #riseAgainstBigTech #reticulum
@papiris what you are asking for is....kubernetes. offload state to some other machine if possible (db, storage...) and use some dedicated OS like Talos and reduce the amount of maintenance. argo or flux for gitops, for talos talhelper is very nice and declarative.
if you can live with giving up some features, have a look at incus or proxmox. but if you want storage with high availability you are probably going to use ceph and the complexity increases.
-
@papiris what you are asking for is....kubernetes. offload state to some other machine if possible (db, storage...) and use some dedicated OS like Talos and reduce the amount of maintenance. argo or flux for gitops, for talos talhelper is very nice and declarative.
if you can live with giving up some features, have a look at incus or proxmox. but if you want storage with high availability you are probably going to use ceph and the complexity increases.
@monotux
regarding storage: ceph is again, quite complex. alternatively, you can get pretty far with zfs snapshots and replication (https://klarasystems.com/articles/zfs-vs-ceph-do-you-actually-need-ceph/).
i'll also be deploying garage soon for s3 compat distributed objstore, which is built to be run by ppl without datacenters (https://garagehq.deuxfleurs.fr/)
@papiris -
@happyborg
imv, proxmox overreaches by taking control of the OS layer. no, thank you, i want to run and completely control that part (debian stable with my own admin processes). #incus is a better, more flexible, simpler option for the virt/container layer.
@shivjm @papiris -
Kubernetes (and k3s) is a damn headache.
So many moving parts and pitfalls.
So easy and quick to seriously mess up.
So difficult and time-consuming to repair.Thousands of hours of my life spent babying a kubernetes homelab. Little actual usefulness produced. I don't want this kind of life.
Instead I want to put less time and effort towards tending digital infra, and more time towards organizing community around that digital infra; so our communities can move away from big tech.Do y'all have suggestions for FLOSS k3s alternatives?
For my use case, it needs to:
- run distributed on a cluster of old linux computers
- run linux containers
- be able to schedule workloads to specific nodes, and reschedule if the node drops off
- be resilient to nodes dropping off and rejoining (both cluster, apps and databases should be fine)
- have deterministic configuration / use git as a source of truth for cluster config and app config. I like the nix method (compute derivation in isolation, only deploy if it will work, immutable) better than the ansible method (commit changes directly to the live system, try to roll back if something breaks. Mutable)
- be easier than k3s
- be more resilient to cluster admin screwups than k3sNice to have:
- self-healing
- simple disaster recovery
- geo-distribution
- delay-tolerance
- tolerance towards low inter-node bandwidth
- ability to use transport-agnostic p2p networking (like reticulum)Your suggestions are very welcome
(Meme nabbed from https://lemmy.world/post/13302376)
#fedihelp #kubernetes #k3s #coopcloud #incus #homelab #IT #techcoop #riseAgainstBigTech #reticulum
@papiris I got a bit fed up with ansible today. Have you heard of mgmt? https://mgmtconfig.com/
I only looked at the documentation and a few examples. It's "let's mutate the machine" like ansible, but you have a proper programming language for it with proper dependency graphs. Seems much more sane to me. I might experiment with it and consider moving data.coop over to it.
