[RFC] Multi-Tenant Workload Identity

[matheuscscp]

In this RFC we aim to add support for multi-tenant workload identity in Flux, i.e. the ability to specify at the object-level which set of cloud provider permissions must be used for interacting with the respective cloud provider on behalf of the reconciliation of the object. In this process, credentials must be obtained automatically, i.e. this feature must not involve the use of secrets. This would be useful in a number of Flux APIs that need to interact with cloud providers, including all controllers except helm-controller.

Preview: https://github.com/fluxcd/flux2/blob/rfc-multi-tenant-workload-identity/rfcs/0010-multi-tenant-workload-identity/README.md

Umbrella issue for implementation: #5022

[matheuscscp]

I'm now addressing the offline comments I got during KubeCon EU 2025.

From @stealthybox:

An alternative to using service account tokens would be using a token whose subject string encodes a direct reference to the respective Flux resource, this way a resource would be its own identity. This is more secure than having a configuration knob to define another resource (a service account in this case) as the object identity, as it prevents another object in the same namespace from abusing the same service account/cloud permissions.

From @hiddeco:

We should move the interfaces in interfaces.go to multiple files with names matching the interface names themselves.

From @stefanprodan:

To avoid introducing kustomizations.spec.decryption.key.serviceAccountName for disambiguating single-tenant and multi-tenant workload identity we should instead introduce a binary flag in the controllers to switch to and enforce multi-tenant workload identity, e.g. --require-service-account-for-provider-auth

[matheuscscp]

@stealthybox @hiddeco @stefanprodan Comments from KubeCon addressed, please feel free to do another pass/approve.

[pjbgf]

LGTM

[hiddeco]

🚀

[matheuscscp]

Putting it like this, it actually doesn't sound like a bad idea after all. Why would you want to use multi-tenancy for Kubernetes RBAC but not for cloud? Or the opposite? It's hard for me to imagine a decent use case where multi-tenancy should apply only to one of those two. That said, users currently using multi-tenancy for RBAC would suddenly be forced to use it also for cloud if we used the same field for both things, and that would be a really bad breaking change.

With all of this in mind, if there is no justifiable use case of multi-tenancy only for RBAC or only for cloud, I propose we use the existing spec.serviceAccountName and existing --default-service-account, and introduce also a feature gate to avoid the breaking change, e.g. UseServiceAccountForDecryption. This feature gate can start opt-in and then change to opt-out later.

But I'm not so much hell bent on this idea, if we can find arguments or use cases of multi-tenancy only for RBAC or only for cloud I'm totally fine with giving up on this (the existing use cases of people using multi-tenancy for RBAC do not count, my point is that they should be using multi-tenancy for cloud as well if it was supported)

[matheuscscp]

Hmmm but people could be using different cloud identities for decryption through secrets and not be using multi-tenant RBAC... Would that justify multi-tenancy for cloud but not for RBAC? 🤔 🤔 🤔

[matheuscscp]

@stefanprodan I changed this part, please review. I think spec.stsEndpoint is better for the other APIs, spec.sts makes sense only for Bucket