Running Talos inside Talos for multi tenancy

[ath88]

Hello,

I imagine this question has come up before, but I can't seem to find it with my search foo.

We have some huge metal servers running a great Talos cluster (tier 1), and we would like to supply smaller clusters to our tenants (tier 2).

Now - to supply isolated Talos clusters to our tenants, we are looking at two options:

1. Running Talos pods directly in the tier 1 cluster, similarly to how Talos runs locally via docker.
2. Using KubeVirt to deploy additional Talos nodes inside of our tier 1 cluster.

I haven't seen anyone using option 1, so I wonder if there's some apparent downside to this.

Option 2 seems quite doable - but I imagine it comes with additional overhead compared to option 1.

We have been looking at vCluster, but the pricing makes it unattractive.

Please talk me out of option 1, so I can stop wasting my time on it. :)

[rothgar]

Option 1 is a mismatch between Talos as a machine operating system and a container process. Talos assumes it can do things like reboot and relies on things like SMBIOS for a unique identity. There are also some things like system extensions that don't work inside containers.

Can you do it? Yes. Some options like vCluster and Kamaji do.
Are there limitations? Yes.

We are making provisioning KubeVirt based clusters easy (or at least easier) with Omni providers. You will be able to run a provider that will call to an infrastructure provider. These can be anything that provide infrastructure resources (eg KubeVirt, VMware, Bare Metal, AWS)

The idea is you'll be able to run a K8s+KubeVirt cluster or your bare metal and then carve it up into smaller "tier 2" clusters that still get VM isolation for resource sharing and kernel isolation. This should be secure enough for multi-tenancy and hopefully as simple enough as using pods.

It's a bit more work to get started but if you're using Omni the base cluster management and provisioning should be easier than doing manual talos configs.

We don't have all the documentation for this ready, but the provider works today. You can connect it to Omni and provision a cluster and the provider will create the VMs and cluster for you.

[ath88]

Thanks! This provided me with what I needed. I'll focus on KubeVirt and I'll take a look at Omni.

[rothgar]

We should have some more docs and content (blogs, videos) on how it'll work in the coming months. We're still finishing up UX and development on the features.

If you want to email me to explain your use case more I think it would be good to jump on a call. justin.garrison@siderolabs.com