Skip to content

Commit

Permalink
feat: function templates
Browse files Browse the repository at this point in the history
  • Loading branch information
matteo-s committed Dec 6, 2024
1 parent d1a9485 commit 7ab8653
Show file tree
Hide file tree
Showing 9 changed files with 139 additions and 2 deletions.
16 changes: 16 additions & 0 deletions templates/function/dbt.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: dbt-transform
kind: dbt
metadata:
description: 'A simple DBT transformation operation example that converts the
input file (employee table dataitem) to another table using a SQL-based
transformation. Run it with e.g. inputs={"employees": di.key},
and outputs={"output_table": "department-60"})'
version: 91da740dccfc42ee8acf82c1f9165439
labels:
- dbt
- etl
spec:
source:
base64: V0lUSCB0YWIgQVMgKAogICAgU0VMRUNUICAqCiAgICBGUk9NICAgIHt7IHJlZignZW1wbG95ZWVzJykgfX0KKQpTRUxFQ1QgICoKRlJPTSAgICB0YWIKV0hFUkUgICB0YWIuIkRFUEFSVE1FTlRfSUQiID0gJzYwJwo=
lang: sql

18 changes: 18 additions & 0 deletions templates/function/echoservice.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: echoservice
kind: python
metadata:
description: A simple python-based API responding to a POST message with the
"text" field. The function exposes an API using "serve" task.
version: 5cdc18d933df4efb9d157f0c460db724
labels:
- python
- api
spec:
python_version: PYTHON3_10
source:
source: src/serve.py
handler: serve
base64: aW1wb3J0IGpzb24KCmRlZiBpbml0KGNvbnRleHQpOgogICAgcHJpbnQoInNvbWUgaW5pdGlhbGl6YXRpb24gZnVuY3Rpb24iKQogICAgc2V0YXR0cihjb250ZXh0LCAidmFsdWUiLCAic29tZSB2YWx1ZSIpCgpkZWYgc2VydmUoY29udGV4dCwgZXZlbnQpOgoKICAgIGlmIGlzaW5zdGFuY2UoZXZlbnQuYm9keSwgYnl0ZXMpOgogICAgICAgIGJvZHkgPSBqc29uLmxvYWRzKGV2ZW50LmJvZHkpCiAgICBlbHNlOgogICAgICAgIGJvZHkgPSBldmVudC5ib2R5CiAgICBjb250ZXh0LmxvZ2dlci5pbmZvKGYiUmVjZWl2ZWQgZXZlbnQ6IHtib2R5fSIpCiAgICB0ZXh0ID0gYm9keVsidGV4dCJdCgogICAgcmV0dXJuIHsicmVzdWx0IjogZiJoZWxsbyB7dGV4dH0gZnJvbSAne2NvbnRleHQudmFsdWV9JyJ9
lang: python
init_function: init

15 changes: 15 additions & 0 deletions templates/function/llm-serve.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: llm-serving
kind: huggingfaceserve
metadata:
description: Expose OpenAI - compatible API based on a model from HugginFace.
The model should be one of the models/derivatives supported by vLLM or
compatible with the Huggingface transformer library. The examples deploys
Qwen/Qwen2.5-0.5B model as an example.
version: 2184afc12d9e40e593da77c16ddaf969
labels:
- openai
- huggingface
- llm
spec:
model_name: model
path: huggingface://Qwen/Qwen2.5-0.5B
14 changes: 14 additions & 0 deletions templates/function/mlflow-server.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: mlflow-server
kind: mlflowserve
metadata:
name: mlflow-server
description: An Open Inference-compatible API that exposes trained ML Model in
the MLFlow format. The model is specified as a reference to the Model entity
already present in the project.
version: 12a6081daaf640f792ea726786ad255f
labels:
- open inference
- mlflow
spec:
model_name: model
path: store://test1/model/mlflow/model-mlflow
21 changes: 21 additions & 0 deletions templates/function/mlflow-train.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: mlflow_train_model
kind: python
metadata:
description: "Train a cancer classifier ML model using MLFlow framework to
structure and represent the model data. The trained model then can be
deployed with the MLFlowServe framework or used for batch classifications
loading it using the MLFlow framework. "
version: 1cf74099137f462bae03d73f1bbfceb1
labels:
- machine learning
- mlflow
spec:
requirements:
- scikit-learn==1.5.0
- mlflow==2.15.1
source:
lang: python
source: src/train.py
base64: ZnJvbSBkaWdpdGFsaHViX3J1bnRpbWVfcHl0aG9uIGltcG9ydCBoYW5kbGVyCgpmcm9tIGRpZ2l0YWxodWIgaW1wb3J0IGZyb21fbWxmbG93X3J1bgppbXBvcnQgbWxmbG93Cgpmcm9tIHNrbGVhcm4gaW1wb3J0IGRhdGFzZXRzLCBzdm0KZnJvbSBza2xlYXJuLm1vZGVsX3NlbGVjdGlvbiBpbXBvcnQgR3JpZFNlYXJjaENWCgpAaGFuZGxlcigpCmRlZiB0cmFpbihwcm9qZWN0KToKICAgIG1sZmxvdy5za2xlYXJuLmF1dG9sb2cobG9nX2RhdGFzZXRzPVRydWUpCgogICAgaXJpcyA9IGRhdGFzZXRzLmxvYWRfaXJpcygpCiAgICBwYXJhbWV0ZXJzID0geyJrZXJuZWwiOiAoImxpbmVhciIsICJyYmYiKSwgIkMiOiBbMSwgMTBdfQogICAgc3ZjID0gc3ZtLlNWQygpCiAgICBjbGYgPSBHcmlkU2VhcmNoQ1Yoc3ZjLCBwYXJhbWV0ZXJzKQoKICAgIGNsZi5maXQoaXJpcy5kYXRhLCBpcmlzLnRhcmdldCkKICAgIHJ1bl9pZCA9IG1sZmxvdy5sYXN0X2FjdGl2ZV9ydW4oKS5pbmZvLnJ1bl9pZAoKICAgICMgdXRpbGl0eSB0byBtYXAgbWxmbG93IHJ1biBhcnRpZmFjdHMgdG8gbW9kZWwgbWV0YWRhdGEKICAgIG1vZGVsX3BhcmFtcyA9IGZyb21fbWxmbG93X3J1bihydW5faWQpCgogICAgcHJvamVjdC5sb2dfbW9kZWwoCiAgICAgICAgbmFtZT0ibW9kZWwtbWxmbG93IiwKICAgICAgICBraW5kPSJtbGZsb3ciLAogICAgICAgICoqbW9kZWxfcGFyYW1zCik=
handler: train
python_version: PYTHON3_10
17 changes: 17 additions & 0 deletions templates/function/python-etl.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: python-etl
kind: python
metadata:
description: A simple ETL procedure in Python that downloads a csv file and then
returns the result as a dataframe that is stored in the project repository
as table.
version: 8ee36148df634014adf6ab64039477fd
labels:
- python
- etl
spec:
python_version: PYTHON3_10
source:
source: etl.py
handler: download_and_process
base64: aW1wb3J0IHBhbmRhcyBhcyBwZApmcm9tIGRpZ2l0YWxodWJfcnVudGltZV9weXRob24gaW1wb3J0IGhhbmRsZXIKClVSTD0iaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2RhdGFzZXRzL3dvcmxkLWNpdGllcy9yZWZzL2hlYWRzL21haW4vZGF0YS93b3JsZC1jaXRpZXMuY3N2IgoKQGhhbmRsZXIob3V0cHV0cz1bIndvcmxkLWNpdGllcyJdKQpkZWYgZG93bmxvYWRfYW5kX3Byb2Nlc3MoKToKICAgIGRmID0gcGQucmVhZF9jc3YoVVJMKQogICAgZGYucmVzZXRfaW5kZXgoaW5wbGFjZT1UcnVlKQogICAgcmV0dXJuIGRm
lang: python
5 changes: 3 additions & 2 deletions templates/function/python-hello-world.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
name: hello-world
kind: python
metadata:
name: hello-world
description: "Hello world with python: write a string to stdout"
version: 3d64037407ee4c95989a205bb4674cad
labels: []
name: hello-world
labels:
- python
spec:
python_version: PYTHON3_10
source:
Expand Down
14 changes: 14 additions & 0 deletions templates/function/sklearn-server.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: sklearn-server
kind: sklearnserve
metadata:
description: "A Open Inference-compatible API exposing a trained scikit-learn ML
model. The model should already exist and is expected to be found in the
project with the specified key. "
version: 83cb0f2727204b99905f22c7167369de
labels:
- scikit-learn
- service
- open inference
spec:
model_name: model
path: store://test1/model/sklearn/cancer_classifier
21 changes: 21 additions & 0 deletions templates/function/sklearn-train.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: train_cancer_model
kind: python
metadata:
description: A simple example of training a scikit-learn ML model using a
standard dataset available. The function represents a training job that logs
the resulting ML model artifact as Model in the project under
'cancer_classifier' name.
version: 5e064afe3b464b11a3e5c83baa51197e
labels:
- scikit-learn
- machine learning
spec:
requirements:
- scikit-learn==1.2.2
python_version: PYTHON3_10
source:
source: src/train.py
handler: breast_cancer_generator
base64: aW1wb3J0IHBhbmRhcyBhcyBwZApmcm9tIHNrbGVhcm4uZGF0YXNldHMgaW1wb3J0IGxvYWRfYnJlYXN0X2NhbmNlcgoKZnJvbSBkaWdpdGFsaHViX3J1bnRpbWVfcHl0aG9uIGltcG9ydCBoYW5kbGVyCgpmcm9tIHNrbGVhcm4ubW9kZWxfc2VsZWN0aW9uIGltcG9ydCB0cmFpbl90ZXN0X3NwbGl0Cgpmcm9tIHNrbGVhcm4uc3ZtIGltcG9ydCBTVkMKZnJvbSBwaWNrbGUgaW1wb3J0IGR1bXAKaW1wb3J0IHNrbGVhcm4ubWV0cmljcwppbXBvcnQgb3MKCkBoYW5kbGVyKCkKZGVmIGJyZWFzdF9jYW5jZXJfZ2VuZXJhdG9yKHByb2plY3QpOgogICAgIiIiCiAgICBBIGZ1bmN0aW9uIHdoaWNoIGdlbmVyYXRlcyB0aGUgYnJlYXN0IGNhbmNlciBkYXRhc2V0CiAgICAiIiIKICAgIGJyZWFzdF9jYW5jZXIgPSBsb2FkX2JyZWFzdF9jYW5jZXIoKQogICAgZGZfY2FuY2VyID0gcGQuRGF0YUZyYW1lKAogICAgICAgIGRhdGE9YnJlYXN0X2NhbmNlci5kYXRhLCBjb2x1bW5zPWJyZWFzdF9jYW5jZXIuZmVhdHVyZV9uYW1lcwogICAgKQogICAgYnJlYXN0X2NhbmNlcl9sYWJlbHMgPSBwZC5EYXRhRnJhbWUoZGF0YT1icmVhc3RfY2FuY2VyLnRhcmdldCwgY29sdW1ucz1bInRhcmdldCJdKQogICAgZGZfY2FuY2VyID0gcGQuY29uY2F0KAogICAgICAgIFtkZl9jYW5jZXIsIGJyZWFzdF9jYW5jZXJfbGFiZWxzXSwgYXhpcz0xCiAgICApCgogICAgWCA9IGRmX2NhbmNlci5kcm9wKFsndGFyZ2V0J10sYXhpcz0xKQogICAgeSA9IGRmX2NhbmNlclsndGFyZ2V0J10KICAgIFhfdHJhaW4sIFhfdGVzdCwgeV90cmFpbiwgeV90ZXN0ID0gdHJhaW5fdGVzdF9zcGxpdChYLCB5LCB0ZXN0X3NpemUgPSAwLjIwLCByYW5kb21fc3RhdGU9NSkKICAgIHN2Y19tb2RlbCA9IFNWQygpCiAgICBzdmNfbW9kZWwuZml0KFhfdHJhaW4sIHlfdHJhaW4pCiAgICB5X3ByZWRpY3QgPSBzdmNfbW9kZWwucHJlZGljdChYX3Rlc3QpCgogICAgaWYgbm90IG9zLnBhdGguZXhpc3RzKCJtb2RlbCIpOgogICAgICAgIG9zLm1ha2VkaXJzKCJtb2RlbCIpCgogICAgd2l0aCBvcGVuKCJtb2RlbC9jYW5jZXJfY2xhc3NpZmllci5wa2wiLCAid2IiKSBhcyBmOgogICAgICAgIGR1bXAoc3ZjX21vZGVsLCBmLCBwcm90b2NvbD01KQoKICAgIG1ldHJpY3MgPSB7CiAgICAgICAgImYxX3Njb3JlIjogc2tsZWFybi5tZXRyaWNzLmYxX3Njb3JlKHlfdGVzdCwgeV9wcmVkaWN0KSwKICAgICAgICAiYWNjdXJhY3kiOiBza2xlYXJuLm1ldHJpY3MuYWNjdXJhY3lfc2NvcmUoeV90ZXN0LCB5X3ByZWRpY3QpLAogICAgICAgICJwcmVjaXNpb24iOiBza2xlYXJuLm1ldHJpY3MucHJlY2lzaW9uX3Njb3JlKHlfdGVzdCwgeV9wcmVkaWN0KSwKICAgICAgICAicmVjYWxsIjogc2tsZWFybi5tZXRyaWNzLnJlY2FsbF9zY29yZSh5X3Rlc3QsIHlfcHJlZGljdCksCiAgICB9CiAgICBwcm9qZWN0LmxvZ19tb2RlbCgKICAgICAgICAgICAgbmFtZT0iY2FuY2VyX2NsYXNzaWZpZXIiLAogICAgICAgICAgICBraW5kPSJza2xlYXJuIiwKICAgICAgICAgICAgc291cmNlPSIuL21vZGVsLyIsCiAgICAgICAgICAgIG1ldHJpY3M9bWV0cmljcwogICAgKQ==
lang: python

0 comments on commit 7ab8653

Please sign in to comment.