Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating the docs site to use SaladCloud #89

Merged
merged 28 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
38a8fa5
update verbiage to SaladCloud
shawnrushefsky Oct 9, 2024
6b12a12
more instances
shawnrushefsky Oct 9, 2024
203a447
gosh theres a lot
shawnrushefsky Oct 9, 2024
216fd9d
prettier
shawnrushefsky Oct 9, 2024
0db49b9
cleanup blip-with-cog
shawnrushefsky Oct 10, 2024
5edf3ae
clean up batch processing
shawnrushefsky Oct 10, 2024
055d20d
move and name images
shawnrushefsky Oct 10, 2024
3b67ca4
cleanup deployment tutorial
shawnrushefsky Oct 10, 2024
504e080
more spelling and moving images around
shawnrushefsky Oct 10, 2024
df2aad2
link typos
shawnrushefsky Oct 10, 2024
5da8a5f
more, publish some latent content too
shawnrushefsky Oct 10, 2024
6116c7a
fix syntax highlighting
shawnrushefsky Oct 10, 2024
39b8c71
more spelling
shawnrushefsky Oct 10, 2024
aed3f20
could that be it?
shawnrushefsky Oct 10, 2024
e34e55f
prettier
shawnrushefsky Oct 10, 2024
cf95d78
spelling
shawnrushefsky Oct 10, 2024
e4bbff6
Update guides/computer-vision/yolov8/deployment-tutorial.mdx
shawnrushefsky Oct 10, 2024
ffe6cfc
Apply suggestions from code review
shawnrushefsky Oct 10, 2024
5f53441
fix link
shawnrushefsky Oct 10, 2024
54b2f9d
caught another one
shawnrushefsky Oct 10, 2024
b7635d2
a few more
shawnrushefsky Oct 10, 2024
14948a3
dang, missed a bunch
shawnrushefsky Oct 10, 2024
308bfec
redundant cloud
shawnrushefsky Oct 10, 2024
416c7a6
nonstandard punctuation
shawnrushefsky Oct 10, 2024
1d0680b
add cog guide
shawnrushefsky Oct 10, 2024
da9478c
spelling
shawnrushefsky Oct 10, 2024
6a0a99f
white space fix
shawnrushefsky Oct 10, 2024
b5b8e48
language fix
shawnrushefsky Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ In our documentation, we label the "Explanation" section "Products", but otherwi

## Discussing

The easiest way to contribute to the project is by participating in community discussions on the [Salad Discord
The easiest way to contribute to the project is by participating in community discussions on the [SaladCloud Discord
server][discord]. The community often chimes in with helpful advice when you have a question, and you may also find
yourself providing answers and helping others. Be sure to review the [code of conduct][code-of-conduct-page] before
participating.

_Please do not use GitHub issues to ask a question._ We will politely close a GitHub issue that asks a question and
kindly refer you to the Salad Discord server or [official support][support].
kindly refer you to the SaladCloud Discord server or [official support][support].

_Please do not use public GitHub issues to report a security vulnerability._ Instead, refer to our [security
policy][security-page] for responsible disclosure options.
Expand Down
8 changes: 4 additions & 4 deletions .github/SUPPORT.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Support

Refer to the [official support documentation][support] for details on getting in touch with the Salad team.
Refer to the [official support documentation][support] for details on getting in touch with the SaladCloud team.

We also host community discussions on the [Salad Discord server][discord]. The community often chimes in with helpful
advice when you have a question, and you may also find yourself providing answers and helping others. Be sure to review
the [code of conduct][code-of-conduct-page] before participating.
We also host community discussions on the [SaladCloud Discord server][discord]. The community often chimes in with
helpful advice when you have a question, and you may also find yourself providing answers and helping others. Be sure to
review the [code of conduct][code-of-conduct-page] before participating.

[code-of-conduct-page]: ./CODE_OF_CONDUCT.md
[discord]: https://discord.gg/ApSm4Kn7Aq
Expand Down
2 changes: 1 addition & 1 deletion api-specs/s4.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
openapi: 3.0.0
info:
title: Salad Simple Storage Service
title: SaladCloud Simple Storage Service
version: 1.1.0
components:
securitySchemes:
Expand Down
8 changes: 6 additions & 2 deletions cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@
"german",
"italian",
"spanish",
"portuguese"
"portuguese",
"companies",
"filetypes",
"npm"
],
"dictionaryDefinitions": [
{
Expand All @@ -29,5 +32,6 @@
"@cspell/dict-es-es/cspell-ext.json",
"@cspell/dict-pt-br/cspell-ext.json",
"@cspell/dict-pt-pt/cspell-ext.json"
]
],
"allowCompoundWords": true
}
62 changes: 55 additions & 7 deletions dictionaries/salad-cloud.txt
Original file line number Diff line number Diff line change
@@ -1,93 +1,141 @@
-Jxpf
alwayson
azcopy
ASGI
astype
azcopy
btop
buildx
Civitai
CivitAI
ckpt
clid
ComfyUI
comfyui
ComfyUI
containerd
Controlnet
Controlnets
cuda
cudnn
denoise
denoising
distil
devel
dewdney
Diátaxis
distil
dreambooth
dreamshaper
encodedimage
esports
Falco
fastapi
ffprobe
firstphase
Flickr
fooocus
Fooocus
getenv
genpkey
getenv
getsize
gsutil
gevent
gsutil
Healthcheck
healthz
heygordian
httpie
imds
IMDS
imdssdk
imgsz
imshow
infotexts
inpainting
INSTALLDIR
interruptible
italian
iterrows
konieshadow
LAVIS
libgl1-mesa-dev
libsm6
libxext6
libxrender1
listdir
LoRA
LoRAs
Lykon
makedirs
metavoiceio
mintlify
neofetch
nomos8khatLOtf_v20.safetensors
noproxy
notna
numpy
nvcr
Ollama
ONNX
OpenAPI
Pabbly
Pabbly’s
pafy
Pafy
pbtxt
popitem
pretrained
println
proxied
pt
pydantic
pytorch
Pytube
Quickstart
Roboflow
routable
RTMP
rtsp
RTSP
Rundiffusion
s6-overlay-noarch
safetensors
SaladCloud
saladcloudimdssdk
saladcloudimdssdkconfig
saladtechnologies
salesforce
schnell
SD.Next
sdapi
sdnext
SDXL
sdxl
SDXL
sklearn
spanish
subseed
subseeds
threadpoolctl
timeframe
timeframe
tlsv
tmax
tmin
tojson
TPOT
TTFT
ultralytics
Ultralytics
uncond
underperform
unet
upscaler
upscalers
uvicorn
Uvicorn
vaeapp
vCPU
vram
webui
WORKDIR
xformers
xlvaeapp
yolob8m-custom
yolov8
zapier
63 changes: 27 additions & 36 deletions guides/computer-vision/deploy-blip-with-cog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,16 @@ synchronous prediction, streaming output and asynchronous prediction with a webh
impressive, providing different running statuses (STARTING, READY, BUSY and FAILED) within the server.

The Cog prediction server can be further customized to meet specific needs. For instance, its path operation functions
for the health checks and predictions are, by default, declared with async def, running in the same main thread. By
declaring the predictions function with def, a new thread will be spawned to run the prediction when a new request
for the health checks and predictions are, by default, declared with `async def`, running in the same main thread. By
declaring the predictions function with `def`, a new thread will be spawned to run the prediction when a new request
arrives.This setup can prevent long-running synchronous predictions from blocking timely responses to health queries.
Another example is IPv6 support: the server is hardcoded to listen on an IPv4 port, we can modify the code to use IPv6
by replacing 0.0.0.0 with ‘::’ when launching its underlying Uvicorn server.
by replacing `0.0.0.0` with `::` when launching its underlying Uvicorn server.

The [BLIP](https://arxiv.org/pdf/2201.12086) (Bootstrapping Language-Image Pre-training) supports multiple image-to-text
tasks, such as Image Captioning, Visual Question Answering and Image Text Matching. Each task requires a dedicated and
fine-tuned BLIP model that is 1~2 GB in size. We can run inference for the three models of these three tasks
simultaneously on a Salad node that has a GPU with 8GB VRAM.
simultaneously on a SaladCloud node that has a GPU with 8GB VRAM.

[LAVIS](https://github.com/salesforce/LAVIS) (A Library for Language-Vision Intelligence) is the Python deep learning
library, and provides the unified access to the pretrained models, datasets and tasks for multimodal applications,
Expand All @@ -36,29 +36,29 @@ HTTP prediction server on SaladCloud, capable of handling various image-to-text
The following 4 files are necessary for building the image, and we also provide some test code in
[the Github Repo](https://github.com/SaladTechnologies/cog-deploy).

```Text cog.yaml
```yml
build:
gpu: true

predict: "predict.py:Predictor"
predict: 'predict.py:Predictor'
```

The yaml file defines how to build a Docker image and how to run predictions. In this example, We only use the Cog HTTP
prediction server (not its CLI tools), the file is quite simple. When the prediction server is launched, it will read
the file, and then set the number of Uvicorn worker processes to 1 (when the GPU is enabled) and run the provided code -
predict.py for inference.

```Text requirements.txt
```
salesforce-lavis==1.0.2
cog==0.9.8
```

Based on the pytorch base image, we only need to install two Python packages and their dependencies for this
Based on the PyTorch base image, we only need to install two Python packages and their dependencies for this
application.

```Text Dockerfile
```Dockerfile
# Base Image
from docker.io/pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
FROM docker.io/pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime

RUN apt-get update && apt-get install -y curl
RUN pip install --upgrade pip
Expand Down Expand Up @@ -90,9 +90,9 @@ SaladCloud, it can start the inference immediately. Alternatively, the models ca
the container is running. This approach has the advantage of smaller image sizes, allowing for faster builds and pushes.

For the inbound connection, the containers running on SaladCloud need to listen on an IPv6 port. The Cog HTTP prediction
server is currently hardcoded to use an IPv4 port, but this can be easily modified by a sed command in the Dockerfile.
server is currently hardcoded to use an IPv4 port, but this can be easily modified by a `sed` command in the Dockerfile.

```Text predict.py
```python
from cog import BasePredictor, Input, Path
import time
import torch
Expand Down Expand Up @@ -124,7 +124,7 @@ predict(), run the inference based on inputs and return the results.

# Test the image

```Text
```
# Build
docker image build -t docker.io/saladtechnologies/sip:0.0.3-blip -f Dockerfile .

Expand All @@ -140,18 +140,18 @@ docker push docker.io/saladtechnologies/sip:0.0.3-blip

After the container is running, you can log into it and do some tests for health checks and predictions.

<img src="/products/sce/images/bligcog/blipcog1.png" />
<img src="/products/sce/images/bligcog/blipcog2.png" />
<img src="/guides/computer-vision/images/blip-cog1.png" />
<img src="/guides/computer-vision/images/blip-cog2.png" />

The Cog HTTP prediction server is now using IPv6. The port number is configurable via the environment variable - ‘PORT’.

<img src="/products/sce/images/bligcog/blipcog3.png" />
<img src="/guides/computer-vision/images/blip-cog3.png" />

# Deploy the image on SaladCloud

Create a container group with the following parameters:

```Text
```md
# Image Source

saladtechnologies/sip:0.0.3-blip
Expand All @@ -162,33 +162,24 @@ saladtechnologies/sip:0.0.3-blip

# Resource

2 vCPUs, 8GB Memory
Any GPU types with 8 GB or more VRAM
2 vCPUs, 8GB Memory Any GPU types with 8 GB or more VRAM

# Container Gateway

Enbaled, Port 5000
Enabled, Port 5000

# Readiness Probe (Protocol: exec)

Enabled
Protocol: exec
Enabled Protocol: exec

Command: python
Argument1: -c
Argument2: import requests,sys;sys.exit(0 if 'READY' in requests.get('http://[::1]:5000/health-check').text else -1)
Command: python Argument1: -c Argument2: import requests,sys;sys.exit(0 if 'READY' in
requests.get('http://[::1]:5000/health-check').text else -1)

Initial Delay Seconds: 60
Period Seconds: 10
Timeout Seconds: 5
Success Threshold: 1
Failure Threshold: 3
Initial Delay Seconds: 60 Period Seconds: 10 Timeout Seconds: 5 Success Threshold: 1 Failure Threshold: 3

# Environment Variables (Optional)

COG_LOG_LEVEL, INFO (Default) / DEBUG / WARNING
PORT, 5000 (Default) or others

COG_LOG_LEVEL, INFO (Default) / DEBUG / WARNING PORT, 5000 (Default) or others
```

The Readiness Probe is used to evaluate whether a container is ready to accept the traffic from the load balancer. The
Expand All @@ -197,15 +188,15 @@ probe with the protocol - exec, will run the given command inside the container,
Python script is provided here and run regularly to check whether the models have been loaded successfully and the Cog
HTTP prediction server is ready.

<img src="/products/sce/images/bligcog/blipcog6.png" />
<img src="/guides/computer-vision/images/blip-cog6.png" />

# Test the inference endpoint

After the container group is deployed, an access domain name will be created and can be used to access the application.

<img src="/products/sce/images/bligcog/blipcog5.png" />
<img src="/guides/computer-vision/images/blip-cog5.png" />

```Text
```shell
# image_captioning

curl -s -X POST \
Expand Down
Loading