From 77d10be2da6da7271bf4c79e169a3af69d414c17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fl=C3=A1vio=20Ramalho?= Date: Tue, 16 Nov 2021 10:14:33 +0100 Subject: [PATCH] Update docs for release v0.3 Update the documentation to reflect the changes made for the next FuseML release. Also includes fixing some typos and formating. --- README.md | 4 +- docs/CONTRIBUTING.md | 20 +- docs/architecture.md | 2 +- docs/cli.md | 8 +- docs/index.md | 3 +- docs/quickstart.md | 60 +- .../img/kserve-triton-gpu/perf-gpu.yaml | 5242 +++++++++++++++++ .../tutorials/img/kserve-triton-gpu/perf.yaml | 37 +- docs/tutorials/kserve-basic.md | 65 +- docs/tutorials/kserve-triton-gpu.md | 72 +- docs/tutorials/seldon-core.md | 45 +- mkdocs.yml | 4 +- 12 files changed, 5402 insertions(+), 160 deletions(-) create mode 100644 docs/tutorials/img/kserve-triton-gpu/perf-gpu.yaml diff --git a/README.md b/README.md index 1cfc317..65b719b 100644 --- a/README.md +++ b/README.md @@ -14,13 +14,13 @@ We are using the [material theme from squidfunk](https://squidfunk.github.io/mkd Simply run: -``` +```bash docker run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material ``` ## How to build the final docs outside a docker container -``` +```bash pip install --user mkdocs-material mkdocs build ``` diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 9713233..0074374 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -2,8 +2,7 @@ ## Our processes - -This repository is used through the entired development process. All the code is publicly available, and the code reviews are public. **Anyone** is welcome to contribute to those. +This repository is used through the entire development process. All the code is publicly available, and the code reviews are public. **Anyone** is welcome to contribute to those. In terms of design, **anyone** is welcomed to propose requirements. Do not hesitate to share any design requirements or improvements. We will discuss them publicly in this repository. This means that any user, developer, or contributor of an eventual fork is welcomed to propose code or design changes. @@ -24,17 +23,17 @@ The next step is to fork our project: [here](https://docs.github.com/en/github/g Then create your own branch for your bug/feature, edit the code/documentation, commit, and push your changes in your repository ```bash -$ git checkout -b -$ ... -$ git commit -$ git push origin HEAD +git checkout -b +... +git commit +git push origin HEAD ``` Finally, create a pull request (using the web interface or using your favorite github client tool). Expectations for commit and pull requests are expressed below. -If your contribution doesn't land immediately, you might need to rebase your change based on the branch you are targetting with your PR (e.g. `main`): +If your contribution doesn't land immediately, you might need to rebase your change based on the branch you are targeting with your PR (e.g. `main`): ```bash ## Go back to the target branch (e.g. main) branch @@ -58,7 +57,7 @@ $ git push origin HEAD -f - Don't send large new features in a giant commit. - Do not assume the reviewer understands what the original problem was, provide all the necessary tools for reviewers to do their job - Describe why a change is made, not how a change is made. It is encouraged to mention any limitations or problems of the current code, and how the commit is fixing those pains. -- We are following conventional commits (https://www.conventionalcommits.org/en/v1.0.0/). Please write your commit messages appropriately. +- We are following [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). Please write your commit messages appropriately. See also: https://wiki.openstack.org/wiki/GitCommitMessages @@ -100,11 +99,10 @@ We expect the maintainers to be the heralds of the good practices of this docume A few extra rules apply to them: - Maintainers must NOT push to existing branches without going through a PR. -- Maintainers must NOT force merge patches. An exception can be made to unbreak tests. -- A maintainer must regularily triage issues and review PRs +- Maintainers must NOT force merge patches. An exception can be made to fix tests. +- A maintainer must regularly triage issues and review PRs - A maintainer is responsible to enforce the code of conduct. ### Community member behavior Everyone must follow our code of conduct - diff --git a/docs/architecture.md b/docs/architecture.md index 9c6d524..ba91ba8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,3 +1,3 @@ # FuseML Architecture -TBD \ No newline at end of file +TBD diff --git a/docs/cli.md b/docs/cli.md index 94a9400..432eb8c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -160,14 +160,14 @@ services: To register the extension as a YAML file with the FuseML CLI: -``` +```bash > fuseml extension register -f mymlflow.yaml Extension "mymlflow" successfully registered ``` Alternatively, to build the same extension step by step: -``` +```bash > fuseml extension register --id mymlflow -p mlflow --version 1.19.0 --zone dev-server --desc "MLFlow experiment tracking service" Extension "mymlflow" successfully registered @@ -197,7 +197,7 @@ using a prediction service. Getting a workflow to execute is basically a two step process: -- first, configure a workflow definition from a yaml or json file. Consult the [API Refefence](api.md) section for a +- first, configure a workflow definition from a yaml or json file. Consult the [API Reference](api.md) section for a full description of the workflow format. - then, use the command line to assign a codeset to the workflow. The workflow is automatically triggered as a result of the assignment, as well as every time a new version of the assigned codeset is published. The status, as well as the results(s) @@ -303,7 +303,7 @@ steps: To create a workflow from the YAML file with the FuseML CLI: -``` +```bash > fuseml workflow create example.yaml Workflow "example" successfully created ``` diff --git a/docs/index.md b/docs/index.md index 8c723d4..bb60fff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -30,6 +30,7 @@ Well, it depends on what you would you like to do: ## Demo video No time for reading? Here's a sneak preview of the installation process of FuseML components. +
@@ -39,7 +40,7 @@ No time for reading? Here's a sneak preview of the installation process of FuseM The team behind FuseML is reachable through: - [Github](https://github.com/fuseml) -- Slack channel: [Join](https://join.slack.com/t/fuseml/shared_invite/zt-rcs6kepe-rGrMzlj0hrRlalcahWzoWg) +- [Slack](https://join.slack.com/t/fuseml/shared_invite/zt-rcs6kepe-rGrMzlj0hrRlalcahWzoWg) - [YouTube](https://www.youtube.com/channel/UCQLoLTikJDDMXvywWd27FBg) ## Sponsor FuseML diff --git a/docs/quickstart.md b/docs/quickstart.md index 5b2969d..b819c95 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -2,7 +2,7 @@ In order to use FuseML one may simply do the following: -### 1. Install the requirements +## 1. Install Requirements We assume that a Kubernetes cluster is already up and running. If not a good way to start with a local cluster is to use [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/), [Rancher K3s](https://k3s.io/) or [k3d](https://k3d.io/). @@ -16,7 +16,7 @@ Additional software requirements: - _kubectl_: install the package provided by your OS, or check [upstream installation guide](https://kubernetes.io/docs/tasks/tools/#kubectl) -### 2. Download and install the fuseml-installer +## 2. Get `fuseml-installer` ```bash # On Linux @@ -26,13 +26,13 @@ curl -sfL https://raw.githubusercontent.com/fuseml/fuseml/main/install.sh | sh - You should get an output similar to this: ```bash -********* SUCCESS ********** -FuseML Installer is ready! -To start using it just run fuseml-installer --help and enjoy! -**************************** +Verifying checksum... Done. +Preparing to install fuseml-installer into /usr/local/bin +fuseml-installer installed into /usr/local/bin/fuseml-installer +Run 'fuseml-installer --help' to see what you can do with it. ``` -### 3. Fuseml Installer is ready to go so let's install the components on the cluster: +## 3. Install FuseML ```bash fuseml-installer install @@ -47,61 +47,63 @@ You should get an output similar to this: Configuration... 🧭 system_domain: - 🧭 extension_repository: https://raw.githubusercontent.com/fuseml/extensions/main/installer/ + 🧭 extensions_repository: https://raw.githubusercontent.com/fuseml/extensions/main/installer/ + 🧭 force_reinstall: false 🚢 Deploying Istio..... ✔️ Istio deployed -. -✔️ Created system_domain: 172.18.0.2.nip.io +✔️ Created system_domain: 172.18.0.3.nip.io 🚢 Deploying Workloads... ✔️ Workloads deployed -🚢 Deploying Gitea................................. -✔️ Gitea deployed (http://gitea.172.18.0.2.nip.io). +🚢 Deploying Gitea...................................................... +✔️ Gitea deployed (http://gitea.172.18.0.3.nip.io). -🚢 Deploying Registry.......... +🚢 Deploying Registry............... ✔️ Registry deployed -🚢 Deploying Tekton............................................................................. -✔️ Tekton deployed (http://tekton.172.18.0.2.nip.io). +🚢 Deploying Tekton................................................................................ +✔️ Tekton deployed (http://tekton.172.18.0.3.nip.io). -🚢 Deploying Core........... -✔️ FuseML core component deployed (http://fuseml-core.172.18.0.2.nip.io). +🚢 Deploying Core.............. +✔️ FuseML core component deployed (http://fuseml-core.172.18.0.3.nip.io). 🚢 Downloading command line client... -🚢 FuseML command line client saved as /home/jsuchome/kubernetes/fuseml/fuseml. +🚢 FuseML command line client saved as /Users/flaviosr/workspace/fuseml/fuseml. Copy it to the location within your PATH (e.g. /usr/local/bin). 🚢 To use the FuseML CLI, you must point it to the FuseML server URL, e.g.: - export FUSEML_SERVER_URL=http://fuseml-core.172.18.0.2.nip.io + export FUSEML_SERVER_URL=http://fuseml-core.172.18.0.3.nip.io ✔️ FuseML installed. -System domain: 172.18.0.2.nip.io +System domain: 172.18.0.3.nip.io ``` -The FuseML installer prints out URLs for _gitea_, _tekton_ and _fuseml-core_ components. You can access those sub-services directly at any given time. The fuseml core URL is especially important, as you'll need that to use the FuseML CLI. +The FuseML installer prints out URLs for _gitea_, _tekton_ and _fuseml-core_ components. You can access those sub-services directly at any given time. The fuseml-core URL is especially important, as you'll need that to use the FuseML CLI. You can also see that the installer downloaded _fuseml_ command line client which you will use for actual FuseML operations. You can use the latest version which was downloaded automatically or manually choose one of [released versions](https://github.com/fuseml/fuseml-core/releases) from github. -Use +Run the following command to see the list of available commands: ```bash fuseml-installer install --help ``` -to get full information about the options the installer provides. - -### 4. Check if the components are up and running: +## 4. Ensure FuseML components are up and running ```bash -# Let's check the fuseml-core component -kubectl get pods -n fuseml-core -o wide -# Now let's check gitea +# Check Istio +kubectl get pods -n istio-system -o wide +# Check Gitea kubectl get pods -n gitea -o wide -# Let's check tekton +# Check registry +kubectl get pods -n fuseml-registry -o wide +# Check tekton kubectl get pods -n tekton-pipelines -o wide +# Check fuseml-core +kubectl get pods -n fuseml-core -o wide ``` If everything is in running or completed status, you are good to go. Continue on to the [tutorial](tutorials/kserve-basic.md) section and start to have fun with FuseML. diff --git a/docs/tutorials/img/kserve-triton-gpu/perf-gpu.yaml b/docs/tutorials/img/kserve-triton-gpu/perf-gpu.yaml new file mode 100644 index 0000000..eac9e99 --- /dev/null +++ b/docs/tutorials/img/kserve-triton-gpu/perf-gpu.yaml @@ -0,0 +1,5242 @@ +apiVersion: batch/v1 +kind: Job +metadata: + generateName: triton-load-test + labels: + app: triton-load-test +spec: + backoffLimit: 6 + parallelism: 1 + template: + metadata: + labels: + app: triton-load-test + annotations: + sidecar.istio.io/inject: "false" + spec: + restartPolicy: OnFailure + containers: + - args: + - vegeta -cpus=5 attack -duration=1m -rate=600/1s -targets=/var/vegeta/cfg + | vegeta report -type=text + command: + - sh + - -c + image: peterevans/vegeta:latest + imagePullPolicy: Always + name: vegeta + volumeMounts: + - mountPath: /var/vegeta + name: vegeta-cfg + volumes: + - configMap: + defaultMode: 420 + name: triton-vegeta-cfg + name: vegeta-cfg +--- +apiVersion: v1 +data: + cfg: | + POST http://demo-cifar10-mlflow-e2e-gpu.fuseml-workloads.svc.cluster.local/v2/models/demo-cifar10-mlflow-e2e-gpu/infer + @/var/vegeta/payload + payload: | + { + "inputs": [ + { + "name": "conv2d_input", + "shape": [1, 32, 32, 3], + "datatype": "FP32", + "data": [ + [ + [ + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ] + ], + [ + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ] + ], + [ + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ] + ], + [ + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.4274509847164154, + 0.45490196347236633, + 0.45098039507865906 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ] + ], + [ + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.4274509847164154, + 0.45490196347236633, + 0.45098039507865906 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ] + ], + [ + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.30980393290519714, + 0.24705882370471954, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ] + ], + [ + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ] + ], + [ + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.30980393290519714, + 0.24705882370471954, + 0.21960784494876862 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ] + ], + [ + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6039215922355652, + 0.5647059082984924, + 0.5647059082984924 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.46666666865348816, + 0.46666666865348816, + 0.4901960790157318 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.4274509847164154, + 0.45490196347236633, + 0.45098039507865906 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ] + ], + [ + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ] + ], + [ + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.30980393290519714, + 0.24705882370471954, + 0.21960784494876862 + ], + [ + 0.30980393290519714, + 0.24705882370471954, + 0.21960784494876862 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.20392157137393951, + 0.2705882489681244, + 0.21568627655506134 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.13725490868091583, + 0.19607843458652496, + 0.1411764770746231 + ], + [ + 0.10588235408067703, + 0.13725490868091583, + 0.10196078568696976 + ] + ], + [ + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.30980393290519714, + 0.24705882370471954, + 0.21960784494876862 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.1764705926179886, + 0.26274511218070984, + 0.18431372940540314 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ], + [ + 0.16470588743686676, + 0.21960784494876862, + 0.18039216101169586 + ] + ], + [ + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.4274509847164154, + 0.45490196347236633, + 0.45098039507865906 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.46666666865348816, + 0.46666666865348816, + 0.4901960790157318 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ], + [ + 0.2549019753932953, + 0.3137255012989044, + 0.27843138575553894 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.2235294133424759, + 0.27450981736183167, + 0.2666666805744171 + ] + ], + [ + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.6039215922355652, + 0.5647059082984924, + 0.5647059082984924 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.35686275362968445, + 0.3843137323856354, + 0.3686274588108063 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.21568627655506134, + 0.30588236451148987, + 0.21960784494876862 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ], + [ + 0.29019609093666077, + 0.3529411852359772, + 0.29411765933036804 + ] + ], + [ + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ] + ], + [ + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.2705882489681244, + 0.32156863808631897, + 0.3333333432674408 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ] + ], + [ + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ], + [ + 0.46666666865348816, + 0.3843137323856354, + 0.33725491166114807 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.46666666865348816, + 0.46666666865348816, + 0.4901960790157318 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5568627715110779, + 0.6000000238418579, + 0.7058823704719543 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ] + ], + [ + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.729411780834198, + 0.7607843279838562, + 0.8313725590705872 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.42352941632270813, + 0.3490196168422699, + 0.3137255012989044 + ], + [ + 0.49803921580314636, + 0.4117647111415863, + 0.364705890417099 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ] + ], + [ + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.729411780834198, + 0.7607843279838562, + 0.8313725590705872 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.4745098054409027, + 0.43921568989753723, + 0.42352941632270813 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.529411792755127, + 0.45098039507865906, + 0.4156862795352936 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ] + ], + [ + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5058823823928833, + 0.5490196347236633, + 0.6235294342041016 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ] + ], + [ + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.7450980544090271, + 0.7843137383460999, + 0.8784313797950745 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.572549045085907, + 0.615686297416687, + 0.7176470756530762 + ], + [ + 0.6000000238418579, + 0.6235294342041016, + 0.686274528503418 + ] + ], + [ + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ] + ], + [ + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.4274509847164154, + 0.45490196347236633, + 0.45098039507865906 + ], + [ + 0.3960784375667572, + 0.42352941632270813, + 0.43529412150382996 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5843137502670288, + 0.6000000238418579, + 0.6509804129600525 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5411764979362488, + 0.5882353186607361, + 0.6784313917160034 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ], + [ + 0.6196078658103943, + 0.6666666865348816, + 0.7490196228027344 + ] + ], + [ + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.47843137383461, + 0.5137255191802979, + 0.45098039507865906 + ], + [ + 0.46666666865348816, + 0.46666666865348816, + 0.4901960790157318 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.48627451062202454, + 0.45490196347236633 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ] + ], + [ + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.4745098054409027, + 0.49803921580314636, + 0.5490196347236633 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.529411792755127, + 0.5098039507865906, + 0.5215686559677124 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.6039215922355652, + 0.6039215922355652, + 0.615686297416687 + ], + [ + 0.5607843399047852, + 0.5411764979362488, + 0.5607843399047852 + ] + ], + [ + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6313725709915161, + 0.6549019813537598, + 0.6980392336845398 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6039215922355652, + 0.5647059082984924, + 0.5647059082984924 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.6274510025978088, + 0.6392157077789307, + 0.6745098233222961 + ], + [ + 0.5372549295425415, + 0.5529412031173706, + 0.5960784554481506 + ], + [ + 0.5686274766921997, + 0.572549045085907, + 0.5843137502670288 + ], + [ + 0.5490196347236633, + 0.5764706134796143, + 0.6352941393852234 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.5529412031173706 + ] + ], + [ + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.5843137502670288, + 0.5803921818733215, + 0.6117647290229797 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6196078658103943, + 0.6235294342041016, + 0.6392157077789307 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ] + ], + [ + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.5215686559677124, + 0.5254902243614197, + 0.4745098054409027 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ], + [ + 0.6666666865348816, + 0.686274528503418, + 0.7333333492279053 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6745098233222961, + 0.6705882549285889, + 0.6705882549285889 + ] + ], + [ + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ] + ], + [ + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6000000238418579, + 0.5411764979362488, + 0.5215686559677124 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7176470756530762, + 0.7333333492279053, + 0.772549033164978 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ] + ], + [ + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7176470756530762, + 0.7019608020782471, + 0.6784313917160034 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.5058823823928833, + 0.6235294342041016, + 0.4000000059604645 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.5568627715110779, + 0.5490196347236633, + 0.5215686559677124 + ], + [ + 0.529411792755127, + 0.4745098054409027, + 0.45490196347236633 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6235294342041016, + 0.6196078658103943, + 0.6039215922355652 + ], + [ + 0.6117647290229797, + 0.6000000238418579, + 0.5803921818733215 + ], + [ + 0.6980392336845398, + 0.6941176652908325, + 0.6823529601097107 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.572549045085907, + 0.5137255191802979, + 0.4941176474094391 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7372549176216125, + 0.729411780834198, + 0.7176470756530762 + ], + [ + 0.7333333492279053, + 0.7176470756530762, + 0.6980392336845398 + ] + ], + [ + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.5058823823928833, + 0.6235294342041016, + 0.4000000059604645 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.6705882549285889, + 0.6627451181411743, + 0.6431372761726379 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6392157077789307, + 0.5882353186607361, + 0.5686274766921997 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.5764706134796143, + 0.572549045085907, + 0.5490196347236633 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.658823549747467, + 0.6470588445663452, + 0.6352941393852234 + ], + [ + 0.6117647290229797, + 0.6627451181411743, + 0.5372549295425415 + ], + [ + 0.6352941393852234, + 0.6392157077789307, + 0.6392157077789307 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.7137255072593689, + 0.7098039388656616, + 0.7019608020782471 + ], + [ + 0.6941176652908325, + 0.6784313917160034, + 0.658823549747467 + ], + [ + 0.6549019813537598, + 0.6313725709915161, + 0.6117647290229797 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ], + [ + 0.7803921699523926, + 0.7686274647712708, + 0.7568627595901489 + ], + [ + 0.7568627595901489, + 0.7490196228027344, + 0.7372549176216125 + ] + ] + ] + ] + } + ] + } +kind: ConfigMap +metadata: + name: triton-vegeta-cfg diff --git a/docs/tutorials/img/kserve-triton-gpu/perf.yaml b/docs/tutorials/img/kserve-triton-gpu/perf.yaml index 9833690..1cfe3e2 100644 --- a/docs/tutorials/img/kserve-triton-gpu/perf.yaml +++ b/docs/tutorials/img/kserve-triton-gpu/perf.yaml @@ -16,28 +16,28 @@ spec: spec: restartPolicy: OnFailure containers: - - args: - - vegeta -cpus=5 attack -duration=1m -rate=600/1s -targets=/var/vegeta/cfg - | vegeta report -type=text - command: - - sh - - -c - image: peterevans/vegeta:latest - imagePullPolicy: Always - name: vegeta - volumeMounts: - - mountPath: /var/vegeta - name: vegeta-cfg + - args: + - vegeta -cpus=5 attack -duration=1m -rate=600/1s -targets=/var/vegeta/cfg + | vegeta report -type=text + command: + - sh + - -c + image: peterevans/vegeta:latest + imagePullPolicy: Always + name: vegeta + volumeMounts: + - mountPath: /var/vegeta + name: vegeta-cfg volumes: - - configMap: - defaultMode: 420 - name: triton-vegeta-cfg - name: vegeta-cfg + - configMap: + defaultMode: 420 + name: triton-vegeta-cfg + name: vegeta-cfg --- apiVersion: v1 data: cfg: | - POST http://demo-cifar10.fuseml-workloads.svc.cluster.local/v2/models/demo-cifar10/infer + POST http://demo-cifar10-mlflow-e2e.fuseml-workloads.svc.cluster.local/v2/models/demo-cifar10-mlflow-e2e/infer @/var/vegeta/payload payload: | { @@ -5239,5 +5239,4 @@ data: } kind: ConfigMap metadata: - annotations: - name: triton-vegeta-cfg \ No newline at end of file + name: triton-vegeta-cfg diff --git a/docs/tutorials/kserve-basic.md b/docs/tutorials/kserve-basic.md index 7a922e7..51c2c79 100644 --- a/docs/tutorials/kserve-basic.md +++ b/docs/tutorials/kserve-basic.md @@ -1,4 +1,4 @@ -# A simple logistic regression with MLFlow and KServe +# A simple logistic regression with MLflow and KServe This example shows how FuseML can be used to automate and end-to-end machine learning workflow using a combination of different tools. In this case, we have a scikit-learn ML model that is being trained @@ -7,47 +7,46 @@ using [MLflow](https://mlflow.org/) and then served with [KServe](https://github We assume that both FuseML infrastructure and the FuseML CLI are already installed, if not please check first the [quick start](../quickstart.md) section. -**1.** Install 3rd party ML tools +## 1. Install 3rd party ML tools Running this example requires MLflow and KServe to be installed in the same cluster as FuseML. -The FuseML installer can be used for a quick MLFlow and KServe installation: +The FuseML installer can be used for a quick MLflow and KServe installation: ```bash fuseml-installer extensions --add mlflow,kserve ``` -To find out about installed extensions, install new or remove installed ones, use +Run the following command to see the list of available commands regarding extensions, +such as listing/removing installed extensions: ```bash fuseml-installer extensions --help ``` -command. - Alternatively, you can follow [the KServe official instructions](https://github.com/kserve/kserve#readme) and install KServe manually. -**2.** Set `FUSEML_SERVER_URL` environment variable to point to fuseml-core +## 2. Set `FUSEML_SERVER_URL` environment variable -The fuseml-core URL was printed out by the installer during the FuseML installation.Alternatively, you can run the following -command to retrieve the fuseml-core URL and set the `FUSEML_SERVER_URL` environment variable: +The fuseml-core URL was printed out by the installer during the FuseML installation. Alternatively, the +following command can be used to retrieve the fuseml-core URL and set the `FUSEML_SERVER_URL` environment variable: ```bash export FUSEML_SERVER_URL=http://$(kubectl get VirtualService -n fuseml-core fuseml-core -o jsonpath="{.spec.hosts[0]}") ``` -**3.** Fetch the FuseML examples code +## 3. Fetch the FuseML examples code ```bash -git clone --depth 1 -b release-0.1 https://github.com/fuseml/examples.git +git clone --depth 1 -b release-0.3 https://github.com/fuseml/examples.git cd examples ``` Under the `codesets/mlflow` directory, there are some example MLflow projects. For this tutorial we will be using the `sklearn` project. -**4.** Register the codeset +## 4. Register the codeset From now on, you start using _fuseml_ command line tool. Register the example code as a FuseML versioned codeset artifact: @@ -68,9 +67,9 @@ Setting mlflow-project-01 as current project. FuseML configuration file created at /home/snica/.config/fuseml/config.yaml ``` -You may optionally log into the Gitea UI using the URL, username and password printed out by the `codeset register` command. You should find a new organization named `mlflow-project-01` and a repo named `mlflow-test`. +You may optionally log into the Gitea UI using URL, username and password printed out by the `codeset register` command. You should find a new organization named `mlflow-project-01` and a repository named `mlflow-test`. -**5.** Create a workflow +## 5. Create a workflow The example FuseML workflow included in the examples repository represents a complete, end-to-end ML pipeline "compatible" with any codeset that contains an MLProject. It includes all the steps necessary to train a model with MLflow, save the model and then creates a KServe prediction service for it. @@ -80,13 +79,13 @@ Use the example workflow definition to create a workflow in FuseML: fuseml workflow create workflows/mlflow-e2e.yaml ``` -**6.** Assign the codeset to the workflow +## 6. Assign the workflow to the codeset ```bash fuseml workflow assign --name mlflow-e2e --codeset-name mlflow-test --codeset-project mlflow-project-01 ``` -**7.** Monitor the workflow from the command-line +## 7. Monitor the workflow from the command-line Now that the Workflow is assigned to the Codeset, a new workflow run was created. To watch the workflow progress, check "workflow run" with: @@ -96,7 +95,7 @@ fuseml workflow list-runs --name mlflow-e2e Example output: -``` +```bash > fuseml workflow list-runs --name mlflow-e2e +--------------------------------------------+------------+--------------+----------+---------+ | NAME | WORKFLOW | STARTED | DURATION | STATUS | @@ -107,10 +106,10 @@ Example output: This command shows you detailed information about running workflow. You may also follow the Tekton URL value under the expanded output section to see relevant information about the underlying Tekton PipelineRun which implements the workflow run: -``` +```yaml > fuseml workflow list-runs --name mlflow-e2e --format yaml --- -- name: fuseml-mlflow-project-01-mlflow-test-hr67k +- name: fuseml-mlflow-project-01-mlflow-test-mlprp workflowref: mlflow-e2e inputs: - input: @@ -119,7 +118,7 @@ This command shows you detailed information about running workflow. You may also type: codeset default: null labels: [] - value: http://gitea.10.162.66.101.omg.howdoi.website/mlflow-project-01/mlflow-test.git:main + value: http://gitea.172.18.0.2.nip.io/mlflow-project-01/mlflow-test.git:main - input: name: predictor description: type of predictor engine @@ -136,12 +135,12 @@ This command shows you detailed information about running workflow. You may also starttime: 2021-06-07T18:16:00Z completiontime: null status: Running - url: "http://tekton.10.162.66.101.omg.howdoi.website/#/namespaces/fuseml-workloads/pipelineruns/fuseml-mlflow-project-01-mlflow-test-hr67k" + url: "http://tekton.172.18.0.2.nip.io/#/namespaces/fuseml-workloads/pipelineruns/fuseml-mlflow-project-01-mlflow-test-mlprp" ``` Once the run succeeds, the status value changes to `Succeeded` in the CLI: -``` +```bash > fuseml workflow list-runs --name mlflow-e2e +--------------------------------------------+------------+----------------+------------+-----------+ | NAME | WORKFLOW | STARTED | DURATION | STATUS | @@ -150,7 +149,7 @@ Once the run succeeds, the status value changes to `Succeeded` in the CLI: +--------------------------------------------+------------+----------------+------------+-----------+ ``` -**8.** Test the deployed model +## 9. Test the deployed model When the workflow run is complete, a new inference service is created and registered as a FuseML application. You can list the applications deployed by FuseML by running the following command: @@ -161,16 +160,16 @@ fuseml application list This should produce output similar to this one: -``` +```bash > fuseml application list -+-------------------------------+-----------+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+------------+ -| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | -+-------------------------------+-----------+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+------------+ -| mlflow-project-01-mlflow-test | predictor | Application generated by mlflow-e2e workflow | http://mlflow-project-01-mlflow-test.fuseml-workloads.10.162.66.101.omg.howdoi.website/v2/models/mlflow-project-01-mlflow-test/infer | mlflow-e2e | -+-------------------------------+-----------+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------+------------+ ++------------------------------------------+-----------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+------------+ +| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | ++------------------------------------------+-----------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+------------+ +| mlflow-project-01-mlflow-test-mlflow-e2e | predictor | Application generated by mlflow-e2e workflow | http://mlflow-project-01-mlflow-test-mlflow-e2e.fuseml-workloads.172.18.0.2.nip.io/v2/models/mlflow-project-01-mlflow-test-mlflow-e2e/infer | mlflow-e2e | ++------------------------------------------+-----------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+------------+ ``` -The application URL is used to submit a request to the inference service. The examples repository includes a prediction data sample at +The application URL can be used to submit a request to the inference service. The examples repository includes a prediction data sample at `prediction/data-sklearn.json`: ```bash @@ -216,7 +215,7 @@ The output will be something similar to: } ``` -**9.** (Optional) Use the webapp example to test the deployed model +## 10. (Optional) Use the webapp example to test the deployed model Rather than using curl to exercise the inference service, you may use a simple app we developed using [streamlit](https://streamlit.io/). @@ -237,8 +236,8 @@ For example: ```bash ❯ kubectl get ksvc -n fuseml-workloads winery -NAME URL LATESTCREATED LATESTREADY READY REASON -winery http://winery.fuseml-workloads.fuseml-workloads.10.162.66.101.omg.howdoi.website winery-00001 winery-00001 True +NAME URL LATESTCREATED LATESTREADY READY REASON +winery http://winery.fuseml-workloads.172.18.0.2.nip.io winery-00001 winery-00001 True ``` Open the application URL and follow the instructions presented on the webpage to make your own predictions with the FuseML application you just deployed. diff --git a/docs/tutorials/kserve-triton-gpu.md b/docs/tutorials/kserve-triton-gpu.md index 5653778..775c257 100644 --- a/docs/tutorials/kserve-triton-gpu.md +++ b/docs/tutorials/kserve-triton-gpu.md @@ -428,15 +428,16 @@ k3d cluster create fuseml --image=k3s:v1.21.5-k3s1-cuda --gpus=1 --agents 2 --k3 2. To install FuseML run the following command, note that we are also installing the mlflow and kserve extensions which will be used for model tracking and serving respectively: - ```bash - fuseml-installer install --extensions mlflow,kserve - ``` + ```bash + fuseml-installer install --extensions mlflow,kserve + ``` - !!! note - FuseML will automatically assign a domain based on the cluster load balancer IP address in the format - `.nip.io`. If you want to use a different domain, you can use the `--system-domain` flag. In the - rest of this tutorial we will use `192.168.86.74.nip.io` as the domain. Any reference to `` - should be replaced by your domain. + !!! note + + FuseML will automatically assign a domain based on the cluster load balancer IP address in the format + `.nip.io`. If you want to use a different domain, you can use the `--system-domain` flag. In the + rest of this tutorial we will use `192.168.86.74.nip.io` as the domain. Any reference to `` + should be replaced by your domain. ## Training & Serving the model @@ -492,11 +493,11 @@ The following steps describe how to use FuseML to train the model and serve it. ```bash $ fuseml workflow list-runs - +---------------------------+------------+----------------+----------+---------+ - | NAME | WORKFLOW | STARTED | DURATION | STATUS | - +---------------------------+------------+----------------+----------+---------+ - | fuseml-demo-cifar10-tkgls | mlflow-e2e | 14 seconds ago | --- | Running | - +---------------------------+------------+----------------+----------+---------+ + +--------------------------------------+------------+----------------+----------+---------+ + | NAME | WORKFLOW | STARTED | DURATION | STATUS | + +--------------------------------------+------------+----------------+----------+---------+ + | fuseml-demo-cifar10-mlflow-e2e-tkgls | mlflow-e2e | 14 seconds ago | --- | Running | + +--------------------------------------+------------+----------------+----------+---------+ ``` You can also see a more detailed view of the workflow run through the tekton dashboard, which should @@ -527,11 +528,11 @@ FuseML applications: ```bash $ fuseml application list -+--------------+-----------+----------------------------------------------+----------------------------------------------------------------------------------------+------------+ -| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | -+--------------+-----------+----------------------------------------------+----------------------------------------------------------------------------------------+------------+ -| demo-cifar10 | predictor | Application generated by mlflow-e2e workflow | http://demo-cifar10.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10/infer | mlflow-e2e | -+--------------+-----------+----------------------------------------------+----------------------------------------------------------------------------------------+------------+ ++-------------------------+-----------+--------------------------------------------------------------------+----------------------------------------------------------------------------------------+------------+ +| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | ++-------------------------+-----------+----------------------------------------------+--------------------------------------------------------------------------------------------------------------+------------+ +| demo-cifar10-mlflow-e2e | predictor | Application generated by mlflow-e2e workflow | http://demo-cifar10-mlflow-e2e.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10-mlflow-e2e/infer | mlflow-e2e | ++-------------------------+-----------+----------------------------------------------+--------------------------------------------------------------------------------------------------------------+------------+ ``` The list of FuseML applications include a URL to query the model for predictions. As FuseML is using the @@ -559,9 +560,9 @@ following deer image: Run the following command to send the request to the application URL: ```bash -$ curl -sX POST http://demo-cifar10.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10/infer -d @examples/prediction/data-keras.json | jq +$ curl -sX POST http://demo-cifar10-mlflow-e2e.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10-mlflow-e2e/infer -d @examples/prediction/data-keras.json | jq { - "model_name": "demo-cifar10", + "model_name": "demo-cifar10-mlflow-e2e", "model_version": "1", "outputs": [ { @@ -704,12 +705,12 @@ Now that the model is validated, we can proceed to train and serve the model wit ```bash $ fuseml workflow list-runs - +---------------------------+----------------+----------------+------------+-----------+ - | NAME | WORKFLOW | STARTED | DURATION | STATUS | - +---------------------------+----------------+----------------+------------+-----------+ - | fuseml-demo-cifar10-pld47 | mlflow-e2e-gpu | 8 seconds ago | --- | Running | - | fuseml-demo-cifar10-tkgls | mlflow-e2e | 23 minutes ago | 12 minutes | Succeeded | - +---------------------------+----------------+----------------+------------+-----------+ + +------------------------------------------+----------------+----------------+------------+-----------+ + | NAME | WORKFLOW | STARTED | DURATION | STATUS | + +------------------------------------------+----------------+----------------+------------+-----------+ + | fuseml-demo-cifar10-mlflow-e2e-gpu-pld47 | mlflow-e2e-gpu | 8 seconds ago | --- | Running | + | fuseml-demo-cifar10-mlflow-e2e-tkgls | mlflow-e2e | 23 minutes ago | 12 minutes | Succeeded | + +------------------------------------------+----------------+----------------+------------+-----------+ ``` If you head to the tekton dashboard (`http://tekton.`) and look at the logs for the @@ -747,16 +748,17 @@ List the FuseML applications: ```bash $ fuseml application list -+--------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------+----------------+ -| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | -+--------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------+----------------+ -| demo-cifar10 | predictor | Application generated by mlflow-e2e-gpu workflow | http://demo-cifar10.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10/infer | mlflow-e2e-gpu | -+--------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------+----------------+ ++-----------------------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------------------------------------+----------------+ +| NAME | TYPE | DESCRIPTION | URL | WORKFLOW | ++-----------------------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------------------------------------+----------------+ +| demo-cifar10-mlflow-e2e-gpu | predictor | Application generated by mlflow-e2e-gpu workflow | http://demo-cifar10-mlflow-e2e-gpu.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10-mlflow-e2e-gpu/infer | mlflow-e2e-gpu | +| demo-cifar10-mlflow-e2e | predictor | Application generated by mlflow-e2e workflow | http://demo-cifar10-mlflow-e2e.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10-mlflow-e2e/infer | mlflow-e2e | ++-----------------------------+-----------+--------------------------------------------------+----------------------------------------------------------------------------------------------------------------------+----------------+ ``` Note that we have the same application, but now it is using the new model and is also using the GPU. To confirm that, check the inference service logs at -`http://kserve-web-app./details/fuseml-workloads/demo-cifar10`, +`http://kserve-web-app./details/fuseml-workloads/demo-cifar10-mlflow-e2e-gpu`, it should have something like: ```bash @@ -766,9 +768,9 @@ I1019 12:58:00.349814 1 metrics.cc:290] Collecting metrics for GPU 0: NVIDIA XXX Now, just like before, send a request to the application URL. ```bash -$ curl -sX POST http://demo-cifar10.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10/infer -d @examples/prediction/data-keras.json | jq +$ curl -sX POST http://demo-cifar10-mlflow-e2e-gpu.fuseml-workloads.192.168.86.74.nip.io/v2/models/demo-cifar10-mlflow-e2e-gpu/infer -d @examples/prediction/data-keras.json | jq { - "model_name": "demo-cifar10", + "model_name": "demo-cifar10-mlflow-e2e-gpu", "model_version": "1", "outputs": [ { @@ -805,7 +807,7 @@ We can now benchmark the inference service running which uses the GPU to compute Run the following command again to benchmark the inference service, you can ignore the (`AlreadyExists` error): ```bash -kubectl create -f https://raw.githubusercontent.com/fuseml/docs/main/docs/tutorials/img/kserve-triton-gpu/perf.yaml +kubectl create -f https://raw.githubusercontent.com/fuseml/docs/main/docs/tutorials/img/kserve-triton-gpu/perf-gpu.yaml ``` Wait for the Job to complete and check the logs for the results: diff --git a/docs/tutorials/seldon-core.md b/docs/tutorials/seldon-core.md index 78eeb45..6ddde34 100644 --- a/docs/tutorials/seldon-core.md +++ b/docs/tutorials/seldon-core.md @@ -9,7 +9,7 @@ In fact, we're using the same MLFlow models like in the [kserve tutorial](kserve We assume that both FuseML infrastructure and the FuseML CLI are already installed, if not please check first the [quick start](../quickstart.md) section. -**1.** Install 3rd party ML tools +## 1. Install 3rd party ML tools Running this example requires MLflow and SeldonCore to be installed in the same cluster as FuseML. @@ -27,19 +27,19 @@ fuseml-installer extensions --help command. -**2.** Set `FUSEML_SERVER_URL` environment variable to point to fuseml-core +## 2. Set `FUSEML_SERVER_URL` environment variable -The fuseml-core URL was printed out by the installer during the FuseML installation. Alternatively, you can run the following -command to retrieve the fuseml-core URL and set the `FUSEML_SERVER_URL` environment variable: +The fuseml-core URL was printed out by the installer during the FuseML installation. Alternatively, the +following command can be used to retrieve the fuseml-core URL and set the `FUSEML_SERVER_URL` environment variable: ```bash export FUSEML_SERVER_URL=http://$(kubectl get VirtualService -n fuseml-core fuseml-core -o jsonpath="{.spec.hosts[0]}") ``` -**3.** Fetch the FuseML examples code +## 3. Fetch the FuseML examples code ```bash -git clone --depth 1 https://github.com/fuseml/examples.git +git clone --depth 1 -b release-0.3 https://github.com/fuseml/examples.git cd examples ``` @@ -48,7 +48,7 @@ Under the `codesets/mlflow` directory, there are some example MLflow projects. F **4.** Register the codeset -From now on, you start using *fuseml* command line tool. Register the example code as a FuseML versioned codeset artifact: +From now on, you start using _fuseml_ command line tool. Register the example code as a FuseML versioned codeset artifact: ```bash fuseml codeset register --name "mlflow-test" --project "mlflow-project-01" codesets/mlflow/sklearn @@ -67,9 +67,9 @@ Setting mlflow-project-01 as current project. FuseML configuration file created at /home/snica/.config/fuseml/config.yaml ``` -You may optionally log into the Gitea UI using the URL, username and password printed out by the `codeset register` command. You should find a new organization named `mlflow-project-01` and a repo named `mlflow-test`. +You may optionally log into the Gitea UI using the URL, username and password printed out by the `codeset register` command. You should find a new organization named `mlflow-project-01` and a repository named `mlflow-test`. -**5.** Create a workflow +## 5. Create a workflow The example FuseML workflow included in the examples repository represents a complete, end-to-end ML pipeline "compatible" with any codeset that contains an MLProject. It includes all the steps necessary to train a model with MLflow, save the model and then creates a KFServing prediction service for it. @@ -81,7 +81,7 @@ fuseml workflow create workflows/mlflow-seldon-e2e.yaml Note: you can modify the workflow to fit to your custom needs. Let's take a look at some parameters on the top of workflow description: -``` +```yaml inputs: - name: mlflow-codeset description: an MLFlow compatible codeset @@ -92,17 +92,17 @@ inputs: default: auto ``` -That `predictor` option is set to `auto` value - this means the predictor engine will be set automatically based on the properties +That `predictor` option is set to `auto` value - this means the predictor engine will be set automatically based on the properties of the model. With the model we are using in this tutorial, the prediction value will be set to 'sklearn' value and Seldon Core will use SKLearn server. You could change the default value to `tensorflow` or `triton` to use different inference servers; that is, if your model supports it. -**6.** Assign the codeset to the workflow +## 6. Assign the workflow to the codeset ```bash fuseml workflow assign --name mlflow-seldon-e2e --codeset-name mlflow-test --codeset-project mlflow-project-01 -``` +``` -**7.** Monitor the workflow from the command-line +## 7. Monitor the workflow from the command-line Now that the Workflow is assigned to the Codeset, a new workflow run was created. To watch the workflow progress, check "workflow run" with: @@ -112,7 +112,7 @@ fuseml workflow list-runs --name mlflow-seldon-e2e Example output: -``` +```bash > fuseml workflow list-runs --name mlflow-seldon-e2e +--------------------------------------------+-------------------+---------------+----------+---------+ | NAME | WORKFLOW | STARTED | DURATION | STATUS | @@ -123,8 +123,7 @@ Example output: This command shows you detailed information about running workflow. You may also follow the Tekton URL value under the expanded output section to see relevant information about the underlying Tekton PipelineRun which implements the workflow run: - -``` +```yaml > fuseml workflow list-runs --name mlflow-seldon-e2e --format yaml --- - name: fuseml-mlflow-project-01-mlflow-test-n5xdq @@ -158,7 +157,7 @@ This command shows you detailed information about running workflow. You may also Once the run succeeds, the status value changes to `Succeeded` in the CLI: -``` +```bash > fuseml workflow list-runs --name mlflow-seldon-e2e +--------------------------------------------+-------------------+----------------+------------+-----------+ | NAME | WORKFLOW | STARTED | DURATION | STATUS | @@ -167,7 +166,7 @@ Once the run succeeds, the status value changes to `Succeeded` in the CLI: +--------------------------------------------+-------------------+----------------+------------+-----------+ ``` -**8.** Test the deployed model +## 8. Test the deployed model When the workflow run is complete, a new inference service is created and registered as a FuseML application. You can list the applications deployed by FuseML by running the following command: @@ -178,7 +177,7 @@ fuseml application list This should produce output similar to this one: -``` +```bash > fuseml application list +-------------------------------------------------+-----------+-----------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+ | NAME | TYPE | DESCRIPTION | URL | WORKFLOW | @@ -214,7 +213,7 @@ cat prediction/data-sklearn-seldon.json ``` See that the format of the data sample is different then the one we have used for [kserve example](kserve-basic.md). -It's because with sklearn based example, inference server is set up to communicate with [Seldon Protocol](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#rest-and-grpc-seldon-protocol). Alternatively, +It's because with sklearn based example, inference server is set up to communicate with [Seldon Protocol](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#rest-and-grpc-seldon-protocol). Alternatively, for codesets using tensorflow, seldon-core based inference server would be set up with [tensorflow protocol](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#rest-and-grpc-seldon-protocol). Run the following commands to send a inference request to the deployed model: @@ -230,9 +229,7 @@ The output will be something similar to: { "data": { "names": [], - "ndarray": [ - 6.48634480912767 - ] + "ndarray": [6.48634480912767] }, "meta": { "requestPath": { diff --git a/mkdocs.yml b/mkdocs.yml index 472598a..92e300c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,7 +5,7 @@ nav: - Quick Start: quickstart.md - Tutorials: - Logistic Regression with MLFlow & KServe: tutorials/kserve-basic.md - - Logistic Regression with MLFlow & Seldon Core: tutorials/seldon-core.md + - Logistic Regression with MLFlow & Seldon-Core: tutorials/seldon-core.md - Training & Serving ML Models on GPU with NVIDIA Triton: tutorials/kserve-triton-gpu.md - Benchmarking ML Models on Intel CPUs with Intel OpenVINO: tutorials/openvino-mlflow.md - FuseML Extension Development Use-Case - OpenVINO: tutorials/openvino-extensions.md @@ -44,6 +44,8 @@ markdown_extensions: class: uml-flowchart format: !!python/name:pymdownx.superfences.fence_code_format # TODO: Validate with our legal team. + + - featherlight copyright: "Copyright © 2021 FuseML Author(s)"