Notebooks 2.0 // Controller // Extract All Container Statuses to State

[thesuperzapper]

As raised by @harshad16 in #210, there is an issue with the generateWorkspaceState() method in the Workspace controller.

The issue is that we only look at the main container's status when checking for issues like ImagePullBackOff, but there could be multiple container or initContainers with issues preventing the Pod from becoming ready.

We need to update the following code so that it looks at all the container statuses in the Pod's status.containerStatuses and status.initContainerStatuses and looks for any which are in a "waiting" state with CrashLoopBackOff or ImagePullBackOff reasons, and then aggregate the errors into the stateMessage.

notebooks/workspaces/controller/internal/controller/workspace_controller.go

Lines 1051 to 1076 in 6f14790

	// get container status
	// https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states
	var containerStatus corev1.ContainerStatus
	for _, container := range pod.Status.ContainerStatuses {
	if container.Name == workspacePodTemplateContainerName {
	containerStatus = container
	break
	}
	}
	// get the container state
	containerState := containerStatus.State
	// STATUS: Error (container state)
	if containerState.Waiting != nil {
	if containerState.Waiting.Reason == "CrashLoopBackOff" {
	state = kubefloworgv1beta1.WorkspaceStateError
	stateMessage = stateMsgErrorContainerCrashLoopBackOff
	return state, stateMessage, nil
	}
	if containerState.Waiting.Reason == "ImagePullBackOff" {
	state = kubefloworgv1beta1.WorkspaceStateError
	stateMessage = stateMsgErrorContainerImagePullBackOff
	return state, stateMessage, nil
	}
	}

NOTE: because there could be different reasons for each one being in a waiting state, we should FIRST look for any ImagePullBackOff reasons and then only return all those as the state, then SECOND check for CrashLoopBackOff so we present the most pressing issues to the user in the state first.

[ederign]

/assign @harshad16

[google-oss-prow]

@ederign: GitHub didn't allow me to assign the following users: harshad16.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @harshad16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harshad16]

As discussed at the community meeting, I would like to pick this ticket to work on it.
/assign

[thesuperzapper]

I just wanted to highlight that we made a small change to the Workspace state generation in #240.

The change now extracts Pod events when we cant figure out why a Pod is not running, but its not going to fix the issue listed above, so we still need to handle extracting container status info about non-main containers.