-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Create the initial Helm chart for ZOO-Project-HPGC
ZOO-Project-DRU update the docker image version to the one supporting service logs
- Loading branch information
Showing
49 changed files
with
3,750 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| apiVersion: v2 | ||
| name: zoo-project-hpgc | ||
| description: A Helm chart for Kubernetes | ||
|
|
||
| # A chart can be either an 'application' or a 'library' chart. | ||
| # | ||
| # Application charts are a collection of templates that can be packaged into versioned archives | ||
| # to be deployed. | ||
| # | ||
| # Library charts provide useful utilities or functions for the chart developer. They're included as | ||
| # a dependency of application charts to inject those utilities and functions into the rendering | ||
| # pipeline. Library charts do not define any templates and therefore cannot be deployed. | ||
| type: application | ||
|
|
||
| # This is the chart version. This version number should be incremented each time you make changes | ||
| # to the chart and its templates, including the app version. | ||
| # Versions are expected to follow Semantic Versioning (https://semver.org/) | ||
| version: 0.0.1 | ||
|
|
||
| # This is the version number of the application being deployed. This version number should be | ||
| # incremented each time you make changes to the application. Versions are not expected to | ||
| # follow Semantic Versioning. They should reflect the version the application is using. | ||
| # It is recommended to use it with quotes. | ||
| appVersion: "0.0.4" | ||
|
|
||
| dependencies: | ||
| - name: postgresql | ||
| version: "12.1.9" | ||
| repository: https://charts.bitnami.com/bitnami | ||
| condition: postgresql.enabled | ||
|
|
||
| - name: rabbitmq | ||
| version: "11.5.0" | ||
| repository: https://charts.bitnami.com/bitnami | ||
| condition: broker.enabled | ||
|
|
||
| - name: redis | ||
| version: "17.6.0" | ||
| repository: https://charts.bitnami.com/bitnami | ||
|
|
||
| - name: minio | ||
| version: 12.8.4 | ||
| repository: https://charts.bitnami.com/bitnami | ||
| condition: minio.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # ZOO-Project with DRU, CWL and OGC Application Package for HPC support package | ||
|
|
||
| > The **ZOO-Project** is an open source processing platform, released under MIT/X11 Licence. | ||
| [Overview of ZOO-Project](http://zoo-project.org) | ||
|
|
||
| Trademarks: This software listing is packaged by the ZOO-Project developper team. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement. | ||
|
|
||
| ## Introduction | ||
|
|
||
| This chart bootstraps a [ZOO-Project](http://zoo-project.org) deployment on a cluster using the [Helm](https://helm.sh/) package manager. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * Kubernetes 1.19+ | ||
| * Helm 3.2.0+ | ||
| * PV provisioner support in the underlying infrastructure | ||
|
|
||
| ## Installing the Chart | ||
|
|
||
| To install the chart with the release name `my-zoo-project-hpgc`: | ||
|
|
||
| ```` | ||
| helm repo add zoo-project https://zoo-project.github.io/charts/ | ||
| helm install my-zoo-project-hpgc zoo-project/zoo-project-hpgc --version 0.0.3 | ||
| ```` | ||
|
|
||
| ## Parameters | ||
|
|
||
| ### Global parameters | ||
|
|
||
| | Name | Description | Value | | ||
| |:-------------------------------------------|:---------------------------------------------------------|:-------------------------| | ||
| | global.postgresql.auth.username | User that will be used to connect to postgresql database | zoo | | ||
| | global.postgresql.auth.password | Password for the user | zoo | | ||
| | global.postgresql.auth.database | Database name | zoo | | ||
| | global.postgresql.service.ports.postgresql | PostgreSQL port | "5432" | | ||
|
|
||
| ### PostgreSQL | ||
|
|
||
| | Name | Description | Value | | ||
| |:-------------------------------------------|:---------------------------------------------------------|:-------------------------| | ||
| | postgresql.enabled | Is database used to store process execution status | true | | ||
| | postgresql.primary.initdb.scriptsConfigMap | The init script config map | true | | ||
|
|
||
|
|
||
| ### RabbitMQ | ||
|
|
||
| | Name | Description | Value | | ||
| |:-------------------------------------------|:---------------------------------------------------------|:---------------------------------------------| | ||
| | rabbitmq.auth.username | User that will be used to connect to RabbitMQ | RABBITMQ_USERNAME | | ||
| | rabbitmq.auth.password | Password for the user | CHANGEME | | ||
| | rabbitmq.loadDefinition.enabled | Enable loading a RabbitMQ definitions file to configure RabbitMQ | true | | ||
| | rabbitmq.loadDefinition.existingSecret | Existing secret with the load definitions file | load-definition | | ||
| | rabbitmq.extraConfiguration | Configuration file content: extra configuration to be appended to RabbitMQ configuration | load_definitions = /app/load_definition.json | | ||
|
|
||
| ### MinIO | ||
|
|
||
| | Name | Description | Value | | ||
| |:--------------|:-----------------------------------------------------|:------| | ||
| | minio.enabled | Is MinIO used for storage in place of AWS | false | | ||
|
|
||
| ### CookieCutter | ||
|
|
||
| | Name | Description | Value | | ||
| |:-------------------------|:-----------------------------------------------------|:----------------------------------------------------| | ||
| | cookiecutter.templateUrl | Where to download the cookiecutter from | https://github.com/EOEPCA/proc-service-template.git | | ||
|
|
||
| ### ZOO-Project | ||
|
|
||
| | Name | Description | Value | | ||
| |:---------------------------------|:-------------------------------------------------------------------|:-----------------------------------------------------| | ||
| | zoo.kernel.maincfgtpl | The main.cfg file to use as configuration for the ZOO-Project | "files/zoo-kernel/main.cfg.tpl" | | ||
| | zoo.kernel.oas | The oas.cfg file to use as OpenAPI definitions for the ZOO-Project | "files/zoo-kernel/oas.cfg" | | ||
| | zoo.kernel.htaccess | The htaccess file to store in Apache root folder | "files/zoo-kernel/htaccess" | | ||
| | zoo.openapi.startupsh | The `startUp.sh` script to start the ZOO-FPM | "files/openapi/server/startUp.sh" | | ||
| | zoo.rabbitmq.rabbitmq | The `definition.json` file containing initial RabbitMQ settings | "files/rabbitmq/definitions.json" | | ||
| | zoo.services.cookiecutter_config | The `cookiecutter_config.yaml` file to uses | "files/zoo-services/assets/cookiecutter_config.yaml" | | ||
|
|
||
| ### Workflow | ||
|
|
||
| | Name | Description | Value | | ||
| |:---------------------------------------------------------|:-----------------------------------|:----------------------------------------------------------------------| | ||
| | workflow.storageClass | The storage class used | standard | | ||
| | workflow.defaultVolumeSize | The default volume size allocated | 10190 | | ||
| | workflow.defaultMaxRam | The default max ram allocated | 1024 | | ||
| | workflow.defaultMaxCores | The default max cores allocated | 2 | | ||
| | workflow.calrissianImage | The calrissian image version | "terradue/calrissian:0.12.0" | | ||
| | workflow.imagePullSecrets | ImagePullSecrets is an optional list of references to secrets for the processing namespace to use for pulling any of the images used by the processing pods. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod | [] | | ||
| | workflow.nodeSelector | Constrain on which nodes the processing pods are eligible to run based on the node label | {} | | ||
| | workflow.env | Environmental variables for the processing pods | {} | | ||
| | workflow.additionalInputs.APP | The application name | zoo-project-hpgc | | ||
| | workflow.additionalInputs.STAGEIN_AWS_REGION | The stagein AWS region | RegionOne | | ||
| | workflow.additionalInputs.STAGEIN_AWS_ACCESS_KEY_ID | The stagein AWS access key id | minio-admin | | ||
| | workflow.additionalInputs.STAGEIN_AWS_SECRET_ACCESS_KEY | The stagein AWS secret access key | minio-secret-password | | ||
| | workflow.additionalInputs.STAGEIN_AWS_SERVICE_URL | The stagein AWS service url | http://zoo-project-hpgc-minio.zp.svc.cluster.local:9000 | | ||
| | workflow.additionalInputs.STAGEOUT_AWS_REGION | The stageout AWS region | RegionOne | | ||
| | workflow.additionalInputs.STAGEOUT_AWS_ACCESS_KEY_ID | The stageout AWS access key id | minio-admin | | ||
| | workflow.additionalInputs.STAGEOUT_AWS_SECRET_ACCESS_KEY | The stageout AWS secret access key | minio-secret-password | | ||
| | workflow.additionalInputs.STAGEOUT_AWS_SERVICE_URL | The stageout AWS secret access key | [minio-admin](http://zoo-project-hpgc-minio.zp.svc.cluster.local:9000) | | ||
| | workflow.additionalInputs.STAGEOUT_OUTPUT | The location where to store output | s3://processingresults | | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| class: Workflow | ||
| $namespaces: | ||
| cwltool: http://commonwl.org/cwltool# | ||
| doc: Main stage manager | ||
| id: main | ||
| label: macro-cwl | ||
| inputs: {} | ||
| outputs: | ||
| StacCatalogUri: | ||
| outputSource: | ||
| - node_stage_out/StacCatalogUri | ||
| type: string | ||
| hints: | ||
| "cwltool:Secrets": | ||
| secrets: [] | ||
| requirements: | ||
| SubworkflowFeatureRequirement: {} | ||
| ScatterFeatureRequirement: {} | ||
| InlineJavascriptRequirement: {} |
112 changes: 112 additions & 0 deletions
112
zoo-project-hpgc/files/cwlwrapper-assets/maincwlmetrics.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| class: Workflow | ||
| doc: Main stage manager | ||
| id: stage-manager | ||
| label: theStage | ||
| inputs: | ||
| workflow: | ||
| doc: workflow | ||
| label: workflow | ||
| type: string | ||
| process: | ||
| doc: process | ||
| label: process | ||
| type: string | ||
|
|
||
| outputs: {} | ||
|
|
||
| requirements: | ||
| SubworkflowFeatureRequirement: {} | ||
| ScatterFeatureRequirement: {} | ||
|
|
||
| steps: | ||
|
|
||
| node_metrics_in: | ||
| in: | ||
| inp1: workflow | ||
| inp2: process | ||
| out: | ||
| - results | ||
| run: | ||
| baseCommand: metrics | ||
| hints: | ||
| DockerRequirement: | ||
| dockerPull: terradue/metrics:0.1 | ||
| class: CommandLineTool | ||
| id: clt | ||
| arguments: | ||
| - prefix: --event | ||
| position: 3 | ||
| valueFrom: "started" | ||
| inputs: | ||
| inp1: | ||
| inputBinding: | ||
| position: 1 | ||
| prefix: --workflow | ||
| type: string | ||
| inp2: | ||
| inputBinding: | ||
| position: 2 | ||
| prefix: --process | ||
| type: string | ||
| outputs: | ||
| results: | ||
| type: stdout | ||
| requirements: | ||
| EnvVarRequirement: | ||
| envDef: | ||
| PATH: /srv/conda/envs/env_metrics/bin:/opt/anaconda/bin:/usr/share/java/maven/bin:/opt/anaconda/bin:/opt/anaconda/envs/notebook/bin:/opt/anaconda/bin:/usr/share/java/maven/bin:/opt/anaconda/bin:/opt/anaconda/condabin:/opt/anaconda/bin:/usr/lib64/qt-3.3/bin:/usr/share/java/maven/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin | ||
| ResourceRequirement: { } | ||
|
|
||
| node_stage_in: | ||
| in: | ||
| metrics: node_metrics_in/results | ||
| out: [] | ||
| run: '' | ||
| # | ||
| # on_stage: | ||
| # in: {} | ||
| # out: [] | ||
| # run: '' | ||
|
|
||
| # node_stage_out: | ||
| # in: {} | ||
| # out: [] | ||
| # run: '' | ||
|
|
||
| node_metrics_out: | ||
| in: | ||
| inp1: workflow | ||
| inp2: process | ||
|
|
||
| out: | ||
| - results | ||
| run: | ||
| baseCommand: metrics | ||
| hints: | ||
| DockerRequirement: | ||
| dockerPull: terradue/metrics:0.1 | ||
| class: CommandLineTool | ||
| id: clt | ||
| arguments: | ||
| - prefix: --event | ||
| position: 3 | ||
| valueFrom: "succeeded" | ||
| inputs: | ||
| inp1: | ||
| inputBinding: | ||
| position: 1 | ||
| prefix: --workflow | ||
| type: string | ||
| inp2: | ||
| inputBinding: | ||
| position: 1 | ||
| prefix: --process | ||
| type: string | ||
| outputs: | ||
| results: | ||
| type: stdout | ||
| requirements: | ||
| EnvVarRequirement: | ||
| envDef: | ||
| PATH: /srv/conda/envs/env_metrics/bin:/opt/anaconda/bin:/usr/share/java/maven/bin:/opt/anaconda/bin:/opt/anaconda/envs/notebook/bin:/opt/anaconda/bin:/usr/share/java/maven/bin:/opt/anaconda/bin:/opt/anaconda/condabin:/opt/anaconda/bin:/usr/lib64/qt-3.3/bin:/usr/share/java/maven/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin | ||
| ResourceRequirement: { } |
Oops, something went wrong.