Can Flatcar Container Linux replace BurmillaOS in long term?

[olljanat]

There is only 9 votes in #186 when I'm writing this and lately in my daily work we have been evaluating Flatcar Container Linux which looks promising.

That why I would like to ask community to evaluate Flatcar and tell what what are cons compared to BurmillaOS?
Perhaps those can be solved by contributing that project?

Pros are quite clear:

• Supported by large companies
• Available in all public clouds
• Immutable file system
• Automatic updates

[benok]

Cons:

• Can't customize console

I don't use Kubernetes now; I use docker compose both in my home lab and (a little*) at work.
(*Internal server or public but internal use only server, just <20 users use)

I deploy server VMs both manually and semi-automatically using Terraform.
I'm using BurmillaOS VMs for both dev and prod servers.
(I have to use Windows laptops and I don't like local docker development. I prefer using Esxi or Proxmox.)

So I regularly use dev BurmillaOS VM's console when I set up a new service.

I tried CoreOS several years ago, but I didn't like its dumb console, so I chose RancherOS and the Debian console at that time.

Of course, I can set up a plain Debian VM for development,
but I prefer BurmillaOS's console/system upgradability and more consistent deployment with cloud-init.

I think BurmillaOS fits for home users and small business users like me.

But I also understand Flatcar's all pros.
Though I prefer BurmillaOS, I feel "using Debian for dev and Flatcar for prod" is also promising.

[olljanat]

On BurmillaOS console is just one of the containers.

You can get pretty much same user experience by spinning custom console container on top of Flatcar.

Basically you use need to have SSH server included to it and mount /var/run/docker.sock to it so docker commands works. You can see what is included to console from here

Home users, etc who don’t need newer kernel version and are happy with current features in BurmillaOS, most likely should be able to use v2.0.2 many years still. Eventually there most likely will be some breaking change in Docker which would require kernel update but my guess is that it will take at least 5 years because RedHat Enterprise Linux 8.10 is supported still quite long time and it is using 4.18.x kernels.

[PrplHaz4]

I tried flatcar right before I switched to BurmillaOS, and didn't use it long enough to identify any "real" issues. I decided against it mostly because of how RedHat it felt.

Not having the Debian console, and having to learn/use butane/ignition was more effort than I was willing to put in at the time.

[olljanat]

systemd definitely gives RedHat feelings for me too. I'm not big fan of systemd but most distros has switched to it so I guess that it is time to get familiar with anyway.

When it comes to Ignition, the logic of course is a bit different than in cloud-init but haven't see any showstoppers yet.

However, it is big issue in open source world that there is too many solutions to same problems which slowdown the progress.

Not having the Debian console

This looks to be common topic so perhaps I need to create burmilla/console repository which provides Debian/BurmillaOS console together with Ignition example for Flatcar.

[benok]

It may be a bit off the track but, I think we should consider using systemd enabled base image for the console.
https://github.com/trfore/docker-debian11-systemd

Several packages and commands don't work without systemd recently.
(Especially in Debian12)
Perhaps, we should have systemd for the console.

[olljanat]

It is related in way that because BurmillaOS doesn't use systemd it comes more and more hard to get those packages working.

What we use now is custom init + System Docker (customized version of Docker which stuck in very old version #28).
Rancher added this logic which mounts /sys/fs/cgroup/systemd even when BurmillaOS it selves does not need it but it does not necessarily be enough for applications which expect systemd to be used.

And if we would migrate to systemd and fix some other issues what there is then we would end up very near what Flatcar is.

[olljanat]

btw. Added prototype (without systemd for now) to burmilla/console. Needs a lot of work of course but idea looks to be working so for those who are interested, can try and open issues to that side.

[benok]

Rancher added this logic which mounts /sys/fs/cgroup/systemd even when BurmillaOS it selves does not need it but it does not necessarily be enough for applications which expect systemd to be used.

Thank you for pointing this out.
I've never tried systemd-enabled console on BurmillaOS yet, but I still think it's worth trying.
(It might not work correctly, but it can be better than now.)

I also want to try Flatcar and your custom console prototype.

[srd424]

This is a bit of an obscure option that won't be for most, but I think I'm going to slowly replace Burmilla with NixOS! It's basically one line of extra NixOS config to enable docker (virtualisation.docker.enable = true;), and I've been slowly learning to how to configure and deploy it.

[PrplHaz4]

Docker was included still to Flatcar when I checked last time but to be honest that track have been in hold for a while.

Younger colleagues seems to prefer Ubuntu nowadays regardless the fact that those needs much more maintenance than BurmillaOS so my latest compromise proposal was #186 (reply in thread)

Our internal discussions about that method are still ongoing but I’m definitely interested about community feedback?

BurmillaOS is still delivering for me as-is - small OS surface area to manage with minimal package churn. I haven't run Ubuntu headless in a while but assume it's significantly more hands-on than a typical BurmillaOS setup...Alpine with docker installed may meet my needs but I haven't had any reason to try that experiment yet.

I'm actually surprised BurmillaOS is not more popular given it's so quick to go from a bare system to containers running...

[joshuacox]

And docker as pid 1! That was quite a feat from the original rancherOS team, I am suprprised there is not more support for burmilla myself. I still have a couple of internal VMs running it for some menial tasks. I don't like this maybe being the last release so I'm not likely to expose a burmilla instance publicly at this point.

[olljanat]

I haven't run Ubuntu headless in a while but assume it's significantly more hands-on than a typical BurmillaOS setup...

Normally yes but that is exactly the part which Elemental Toolkit solves by allowing whole OS configuration with cloud-init and upgrades by simply switching to new image version like it is done in BurmillaOS. You should try my new prototype ;)

And docker as pid 1!

That is double-edged sword. Getting things like 3rd party monitoring tools working is almost impossible and never officially supported.

I don't like this maybe being the last release

We have 200+ BurmillaOS nodes running business critical workloads so I definitely need find alternative soon or there needs to be 2.1.0 version with 6.x version kernel. But I’m trying to do better work than Rancher here by not promising it before it is certain.

I'm actually surprised BurmillaOS is not more popular given it's so quick to go from a bare system to containers running...

Nowadays Containers as a Service platforms are good enough so new users goes to there and older user space already has found alternative which works for them.

[joshuacox]

200+ nodes! That is nutz! Have you heard of kubernetes? What are you using to orchestrate such a herd? I'm imagining an australian shepherd running around barking madly at servers running out of resources.

[olljanat]

200+ nodes! That is nutz!

Did you really think that someone would create Linux distro to run it in just couple of nodes?

The whole point of choosing RancherOS in first place was to minimize overhead which allow us to have single purpose nodes which makes implementing changes much easier.

We run Docker Swarm in those. Minimal setup is 3 x manager, 1 x logging and 2 x worker nodes, spread to three different datacenters. Because Docker don’t have proper multi-tenancy each application team+environment combination needs dedicated Swarm.

Have you heard of kubernetes?

Sure, my plan was deprecate BurmillaOS already before 2.0 release so I built multi-tenant Kubernetes cluster with vCluster (and was early contributor in there). On that model each application team+environment combination would get own virtual K8s cluster. It was working fine but teams preferred to focus application development instead this migration.

What are you using to orchestrate such a herd?

That is power of infrastructure as a code, we don’t upgrade nodes, we recreate them when needed.

I'm imagining an australian shepherd running around barking madly at servers running out of resources.

Resource handling in private cloud is actually quite simple as you can just oversize VMs and adjust couple of times per year based on metrics. That why I added https://github.com/burmilla/os/blob/v2.0.4/os-config.tpl.yml#L104-L105 to BurmillaOS in quite early phase that it does not keep memory reserved when containers are not using it.

[joshuacox]

I think my choice at this point if I'm looking for a minimal environment to run a single container, and I don't care to make a helm chart just yet and stick it in a k8s cluster is alpine. The support is great, the footprint is small, installing docker easy. If I run into musl woes, I upgrade to debian instead of fooling with the workarounds in alpine. But for the most part I'm just sticking everything in a k3s cluster these days.

[olljanat]

Thanks about all your feedback.
Closing this discussion because it seems that no one in here (including me) does not like Flatcar Container Linux too much.

Let's continue discussion in Long term planning (year 2027 onwards).