Add an example of using Hamilton in AWS Glue

Author: tyapochkin

examples/aws/glue/README.md

@@ -0,0 +1,88 @@
+[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html) using Hamilton functions on AWS Glue.
+
+## Prerequisites
+
+- **AWS CLI Setup**: Make sure the AWS CLI is set up on your machine. If you haven't done this yet, no worries! You can follow the [Quick Start guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) for easy setup instructions.
+
+## Step-by-Step Guide
+
+### 1. Build wheel with Hamilton functions
+
+First things first, AWS Glue jobs run a single python script, but you can include external code (like our Hamilton functions) by adding it as a python wheel. So, let's package our code and get it ready for action.
+
+- **Install build package:**
+
+    This command installs the 'build' package, which we'll use to create our python wheel.
+
+    ```shell
+    pip install build
+    ```
+
+- **Build python wheel:**
+
+    ```shell
+    cd app && python -m build --wheel --skip-dependency-check && cd ..
+    ```
+
+### 2. Upload all necessary files to S3
+
+- **Upload the wheel file to S3:**
+
+    Replace `<YOUR_PATH_TO_WHL>` with your specific S3 bucket and path:
+
+    ```shell
+    aws s3 cp app/dist/hamilton_functions-0.1-py3-none-any.whl s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl
+    ```
+
+- **Upload main python script to s3:**
+
+    Replace `<YOUR_PATH_TO_SCRIPT>` with your specific S3 bucket and path:
+
+    ```shell
+    aws s3 cp processing.py s3://<YOUR_PATH_TO_SCRIPT>/processing.py
+    ```
+
+- **Upload input data to s3:**
+
+    Replace `<YOUR_PATH_TO_INPUT_DATA>` with your specific S3 bucket and path:
+
+    ```shell
+    aws s3 cp data/input_table.csv s3://<YOUR_PATH_TO_INPUT_DATA>
+    ```
+
+### 3. Create a simple role for AWS Glue job execution
+
+- **Create the Role**:
+
+    ```shell
+    aws iam create-role --role-name GlueProcessorRole --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
+    ```
+
+- **Attach Policies to the Role**:
+
+    Here we grant full access to S3 as an example. For production environments it's important to restrict access appropriately.
+
+    ```shell
+    aws iam attach-role-policy --role-name GlueProcessorRole --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
+    aws iam attach-role-policy --role-name GlueProcessorRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
+    ```
+
+### 4. Create and run the job
+
+- **Create a job:**
+
+    Ensure all paths are correctly replaced with the actual ones:
+
+    ```shell
+    aws glue create-job --name test_hamilton_script --role GlueProcessorRole --command '{"Name" :  "pythonshell", "PythonVersion": "3.9", "ScriptLocation" : "s3://<YOUR_PATH_TO_SCRIPT>/processing.py"}' --max-capacity 0.0625 --default-arguments '{"--extra-py-files" : "s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl", "--additional-python-modules" : "sf-hamilton"}'
+    ```
+
+- **Run the job:**
+
+    Ensure all paths are correctly replaced with the actual ones:
+
+    ```shell
+    aws glue start-job-run --job-name test_hamilton_script --arguments '{"--input-table" : "s3://<YOUR_PATH_TO_INPUT_DATA>", "--output-table" : "s3://<YOUR_PATH_TO_OUTPUT_DATA>"}'
+    ```
+
+    Once you've run the job, you should see an output file at `s3://<YOUR_PATH_TO_OUTPUT_DATA>`.

examples/aws/glue/app/hamilton_functions/__init__.py

@@ -0,0 +1,38 @@
+import pandas as pd
+
+from hamilton.function_modifiers import extract_columns
+
+
+@extract_columns("spend", "signups")
+def raw_table(input_table: pd.DataFrame) -> pd.DataFrame:
+    return input_table
+
+
+def avg_3wk_spend(spend: pd.Series) -> pd.Series:
+    """Rolling 3 week average spend."""
+    return spend.rolling(3).mean()
+
+
+def spend_per_signup(spend: pd.Series, signups: pd.Series) -> pd.Series:
+    """The cost per signup in relation to spend."""
+    return spend / signups
+
+
+def spend_mean(spend: pd.Series) -> float:
+    """Shows function creating a scalar. In this case it computes the mean of the entire column."""
+    return spend.mean()
+
+
+def spend_zero_mean(spend: pd.Series, spend_mean: float) -> pd.Series:
+    """Shows function that takes a scalar. In this case to zero mean spend."""
+    return spend - spend_mean
+
+
+def spend_std_dev(spend: pd.Series) -> float:
+    """Function that computes the standard deviation of the spend column."""
+    return spend.std()
+
+
+def spend_zero_mean_unit_variance(spend_zero_mean: pd.Series, spend_std_dev: float) -> pd.Series:
+    """Function showing one way to make spend have zero mean and unit variance."""
+    return spend_zero_mean / spend_std_dev

examples/aws/glue/app/hamilton_functions/functions.py

@@ -0,0 +1,3 @@
+from setuptools import setup
+
+setup(name="hamilton_functions", version="0.1", packages=["hamilton_functions"])

examples/aws/glue/app/setup.py

@@ -0,0 +1,7 @@
+signups,spend
+1,10
+10,10
+50,20
+100,40
+200,40
+400,50

examples/aws/glue/data/input_table.csv

@@ -0,0 +1,31 @@
+import sys
+
+import pandas as pd
+
+# awsglue is installed in the AWS Glue worker environment
+from awsglue.utils import getResolvedOptions
+from hamilton_functions import functions
+
+from hamilton import driver
+
+if __name__ == "__main__":
+
+    args = getResolvedOptions(sys.argv, ["input-table", "output-table"])
+
+    df = pd.read_csv(args["input_table"])
+
+    dr = driver.Driver({}, functions)
+
+    inputs = {"input_table": df}
+
+    output_columns = [
+        "spend",
+        "signups",
+        "avg_3wk_spend",
+        "spend_per_signup",
+        "spend_zero_mean_unit_variance",
+    ]
+
+    # DAG execution
+    df_result = dr.execute(output_columns, inputs=inputs)
+    df_result.to_csv(args["output_table"])