feat: Publish debug container image and account-sync sidecar #251
Conversation
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
I'm not attached to having this account-sync container built from this repository, but given the recent drop of Debian support for the nss-pam-ldap library in favour of nss-pam-ldapd, and as the configuration is so nestled deep in the inner workings of the container, this seemed a reasonable place for it to be built from to ensure it used a compatible version of the base container image.
There was a problem hiding this comment.
We could also just have a standalone repo in DiamondLightSource for this. It feels like the cadence of release of account-sync vs python-copier-template would be wildly different. Account sync could be built occasionally (almost never after the first time?) and if you update it then you need to update the version tag that copier template refers to.
There was a problem hiding this comment.
I think I prefer that, because otherwise there will be many releases of the account-sync image with the same contents.
There was a problem hiding this comment.
I'll probably end up moving it to its own repository and using some release automation that I want to try out to keep it up to date.
| @@ -14,10 +14,31 @@ ENV PATH=/venv/bin:$PATH{% if docker %} | |||
|
|
|||
| # The build stage installs the context into the venv | |||
| FROM developer AS build | |||
| COPY . /context | |||
| WORKDIR /context | |||
| # Requires buildkit 0.17.0 | |||
There was a problem hiding this comment.
To use chmod with non-octal form
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. | ||
|
|
||
| After the connection to the cluster has been established, it may be necessary to open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer. |
There was a problem hiding this comment.
This does appear to always be necessary: container config files are supported for attaching to generic remote devcontainers, but not currently for those in kubernetes.
There was a problem hiding this comment.
Re kubeconfig. You can select your kubeconfig after launching vscode, see image. The important thing is that kubectl and helm are available on the path. This is challenging when vscode may be launched from gnome (or be already running and attach to the existing process). For this reason I suggest we recommend putting the (current version) tools in the path in .profile like I suggest for docker-compose here https://epics-containers.github.io/main/how-to/compose-quickstart.html#diamond-light-source-workstation. Note that this is peculiar to sites which use 'module load' to get these tools and they are not in the path by default.
| ``` | ||
|
|
||
| The following changes/additions to your `values.yaml` may be required to connect vscode when using the sidecar. | ||
| It is recommended to set the `HOME` environment variable on your container to be debugged to the same value used in the volume below. |
There was a problem hiding this comment.
From my experience, HOME has been unset, so trying to reference ~, or ~/.ssh routes to / or /.ssh respectively. My example-service has HOME being set if debug is enabled.
| volumes: | ||
| - name: home # Required for vscode to start and install plugins | ||
| hostPath: | ||
| path: /home/<fedid> |
There was a problem hiding this comment.
Using just /home/ had issues with "too many levels of symlinks". Not sure if it's just my home dir.
There was a problem hiding this comment.
- which helm chart are we using here?
- do you really need to mount a real home folder? can't we just set HOME to point at folder in the nslcd PVC?
There was a problem hiding this comment.
This is with the chart created by helm create <service>: looking at my little dummy service I've enabled this on, the differences vs. helm create service are: (in Values.yaml) configuring the container image & ingress, (in Deployment.yaml) adding the sidecar, adding a HOME envVar, modifying the tag of the image.
We could use the emptyDir, but if the developer is opinionated about configuration they might want their actual home directory set and then it's an arms race of setting the size of the empty dir. It would allow for an easy to replicate HOME env var. I'm not sure how home for k8s users is configured, so it may be necessary to do that when a service is running as the namespace user.
There was a problem hiding this comment.
I've seen the symlinks problem when mounting automount folders and there is a workaround for this in podman but not sure if that is available to K8S.
I'm not sure you will always be able to mount your own home dir though? If you are not running in your personal namespace then you won't be allowed to run with your own user id. A beamline namespace service account or ixxdetector account would not be allowed to mount your home directory (at least not writeable). Maybe I've missed something here?
I'm afraid I'm away until next Tues now - will discuss on my return.
There was a problem hiding this comment.
Yes AFAIK (and I should update the docs if this isn't clear) the home has to be the home of the user that the pod is running as.
i.e.
podSecurityContext:
runAsUser: <some uid>
runAsGroup: <some gid>
volumeMounts:
- mountPath: /home/<username of the uid>/
name: homeThere was a problem hiding this comment.
OK, I see. I'm pretty sure that shared namespace service accounts don't have a home directory.
I feel that making a PVC and using that to as a home directory for the vscode config would be better. Each developer could make their own if they really wanted and keep it for debugging in the cluster, whichever account the service being debugged requires to run as.
There was a problem hiding this comment.
I feel like that's a slippery slope to re-inventing all of our problems with home directory quotas into the file system.
I think I've justified to myself moving it into the shared volume: the devcontainer is defined from the devcontainer json, not from anything in your home dir. It can be thrown away when the devcontainer isn't required and recreated when the next person needs it.
There was a problem hiding this comment.
I've just tested it and vscode doesn't respect the $HOME environment variable: while the kubernetes plugin does't support container config files (devcontainer.json definitions of cluster attaches) I think mounting something at /home/ may be necessary
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
We could also just have a standalone repo in DiamondLightSource for this. It feels like the cadence of release of account-sync vs python-copier-template would be wildly different. Account sync could be built occasionally (almost never after the first time?) and if you update it then you need to update the version tag that copier template refers to.
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
I think I prefer that, because otherwise there will be many releases of the account-sync image with the same contents.
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. | ||
|
|
||
| After the connection to the cluster has been established, it may be necessary to open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer. |
There was a problem hiding this comment.
Re kubeconfig. You can select your kubeconfig after launching vscode, see image. The important thing is that kubectl and helm are available on the path. This is challenging when vscode may be launched from gnome (or be already running and attach to the existing process). For this reason I suggest we recommend putting the (current version) tools in the path in .profile like I suggest for docker-compose here https://epics-containers.github.io/main/how-to/compose-quickstart.html#diamond-light-source-workstation. Note that this is peculiar to sites which use 'module load' to get these tools and they are not in the path by default.
| volumes: | ||
| - name: home # Required for vscode to start and install plugins | ||
| hostPath: | ||
| path: /home/<fedid> |
There was a problem hiding this comment.
- which helm chart are we using here?
- do you really need to mount a real home folder? can't we just set HOME to point at folder in the nslcd PVC?
| COPY . /context | ||
| WORKDIR /context | ||
| # Requires buildkit 0.17.0 | ||
| COPY --chmod=o+wrX . /workspaces/{{ repo_name }} |
There was a problem hiding this comment.
Do we want to make this all writable for developer as well as debug? It might not matter that much I guess?
|
Overall this is a neat solution - great work.
Other than that my comments were minor. |
DiamondJoseph
changed the title
DiamondJoseph
changed the title
There was a problem hiding this comment.
Thanks, this seems like a good solution to do something quite complicated. A few comments
- As we had a few mysterious issues when trying to get it to work for me, I don't think we can trust the steps in the docs to reliably "just work" yet.
- The details of how this works is pretty confusing for a kube noob, although conceptually "swapping out one container for another and attaching vscode" does make sense
- I agree with @gilesknap 's comment about adding a full helm chart example in the copier template
- To prove this is robust, we should do an iterative process of choosing someone in daq/controls and asking them to try and get this working using only the docs, then adjusting the docs with whatever they were missing
| With the [Kubernetes plugin for vscode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. This may require that your targeted kubeconfig is at `~/.kube/config`, rather than referenced from the environment variable `KUBECONFIG`. It may also be necessary to [add additional contextual information](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_config/kubectl_config_set-context/), such as the namespace. | ||
|
|
||
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. |
There was a problem hiding this comment.
Might be worth mentioning that you need to do a module load {k8s-cluster} in the terminal that you launch vscode from in order for the cluster to appear
|
|
||
| The container build also publishes a debug container for each tagged release of the container with the tag suffixed with `-debug`. This container contains the workspace and has an alternative entrypoint which allows the devcontainer to attach: if you have configured a `livenessProbe` that requires the service to have started it should be disabled. The container also installs debugpy and makes the service install editable. Any custom `command` or `args` defined for the container should be disabled. | ||
|
|
||
| With the [Kubernetes plugin for vscode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. This may require that your targeted kubeconfig is at `~/.kube/config`, rather than referenced from the environment variable `KUBECONFIG`. It may also be necessary to [add additional contextual information](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_config/kubectl_config_set-context/), such as the namespace. |
There was a problem hiding this comment.
Please could you add:
- How do we know if we need to change targeted kubeconfig
- How do we know if its necessary to do the 'add additional contextual information' bit
Closes #234
Provides an account-sync sidecar for current Python base container image to allow vscode to connect to devcontainers running in the cluster as specific users.
Adds a stage to the container build to make a persistent devcontainer with editable code for connecting to.
Adds docs explaining how to configure and connect to containers running in the cluster.
I have made a test repository with Helm configuration and an example service which can be fixed internally to the cluster. I have a PR with commits that I made inside a devcontainer inside the cluster.