ComfyUI on an On-Premise Kubernetes Cluster

This project aims to provide two different types of ComfyUI services on Kubernetes:

1. API Service for Production
2. Interactive Service for Designers

Features

• [Common] Minikube Cluster for development
• [Common] GPU Sharing (Time Slicing)
• [Common] Ingress Route
• [Common] Monitoring (Dashboard for Metric + Logging)
• [ComfyUI API Service] Cookie-based Session Stickiness with Timeout
• [ComfyUI API Service] Python Test Script for Image Generation
• [ComfyUI API Service] Horizontal Pod Autoscaling
• [ComfyUI API Service] Efficient Routing (Max Use of ComfyUI Cache)
• [ComfyUI Interactive Service] Authentication and Authorization
• [ComfyUI Interactive Service] Custom Docker Image for JupyterHub SingleUser (+ComfyUI)
• [ComfyUI Interactive Service] ComfyUI Extension for Jupyter Server Proxy
• [ComfyUI Interactive Service] Profiles for GPU Env / CPU only Env
• [ComfyUI Interactive Service] Persistent Volume for ComfyUI User Data
• [ComfyUI Interactive Service] Evicting inactive users

Prerequisites

• Install Docker & NVIDIA-Container-Toolkit
• Install Kubectl
• Install Minikube
• Install HELM

Cluster Setup

MODEL_PATH=PATH_TO_MODELS make cluster
# Enable GPU Time-Slicing for multiple replicas.
helm install nvidia-device-plugin charts/nvidia-device-plugin -n kube-system
# Install Ingress.
kubectl create namespace ingress
helm install traefik charts/traefik -n ingress

# Volumes.
kubectl apply -f volumes/minikube.yaml  # for minikube env.

ComfyUI Interactive Service (w/ JupyterHub)

Build ComfyUI (Optional)

eval $(minikube docker-env)
make docker-build
make docker-build-jupyter
make docker-push-jupyter
make docker-run-jupyter  # for testing

helm install jupyterhub charts/jupyterhub

• login with id: admin / pw: admin123!@#

ComfyUI API Service

This is an on-premise Kubernetes cluster version inspired by comfyui on EKS.

Build ComfyUI (Optional)

make docker-build
make docker-push
make docker-run  # for testing

helm install comfyui charts/comfyui

• Create a connection for testing in minikube: make tunnel
• open http://localhost/comfyui/
• cd test
• python main.py -s http://host-address/comfyui

Result:

Generation started.
Generation not ready, sleep 1s ...
Generation not ready, sleep 1s ...
Generation finished.
Inference finished.
ClientID: 80e564e7-48ef-44cd-aae0-2a18fa091deb.
PromptID: 8882b193-d7d4-42e4-8003-7cdfe51a465d.
Num of images: 1.
Time spent: 2.17s.
------

References

GPU Sharing on K8S

• https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#with-cuda-time-slicing
• https://nyyang.tistory.com/198

ComfyUI + K8S

• https://github.com/aws-samples/comfyui-on-eks

Ingress

• https://doc.traefik.io/traefik/routing/services/?ref=traefik.io#sticky-sessions
• https://doc.traefik.io/traefik/routing/providers/kubernetes-ingress/

JupyterHub

• https://z2jh.jupyter.org/en/stable/
• https://z2jh.jupyter.org/en/latest/administrator/services.html

Jupyter Server Proxy

• https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html
• https://github.com/jupyterhub/jupyter-rsession-proxy/