Skip to content

Commit 2a5d266

Browse files
tyapochkinskrawcz
authored andcommitted
Reformat all AWS-related README files to have a consistent style
1 parent a2c5235 commit 2a5d266

File tree

3 files changed

+116
-50
lines changed

3 files changed

+116
-50
lines changed

examples/aws/glue/README.md

+32-9
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Deploy Hamilton Functions as an AWS Glue Job
2+
13
[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html) using Hamilton functions on AWS Glue.
24

35
## Prerequisites
@@ -21,7 +23,9 @@ First things first, AWS Glue jobs run a single python script, but you can includ
2123
- **Build python wheel:**
2224

2325
```shell
24-
cd app && python -m build --wheel --skip-dependency-check && cd ..
26+
cd app \
27+
&& python -m build --wheel --skip-dependency-check \
28+
&& cd ..
2529
```
2630

2731
### 2. Upload all necessary files to S3
@@ -31,40 +35,52 @@ First things first, AWS Glue jobs run a single python script, but you can includ
3135
Replace `<YOUR_PATH_TO_WHL>` with your specific S3 bucket and path:
3236

3337
```shell
34-
aws s3 cp app/dist/hamilton_functions-0.1-py3-none-any.whl s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl
38+
aws s3 cp \
39+
app/dist/hamilton_functions-0.1-py3-none-any.whl \
40+
s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl
3541
```
3642

3743
- **Upload main python script to s3:**
3844

3945
Replace `<YOUR_PATH_TO_SCRIPT>` with your specific S3 bucket and path:
4046

4147
```shell
42-
aws s3 cp processing.py s3://<YOUR_PATH_TO_SCRIPT>/processing.py
48+
aws s3 cp \
49+
processing.py \
50+
s3://<YOUR_PATH_TO_SCRIPT>/processing.py
4351
```
4452

4553
- **Upload input data to s3:**
4654

4755
Replace `<YOUR_PATH_TO_INPUT_DATA>` with your specific S3 bucket and path:
4856

4957
```shell
50-
aws s3 cp data/input_table.csv s3://<YOUR_PATH_TO_INPUT_DATA>
58+
aws s3 cp \
59+
data/input_table.csv \
60+
s3://<YOUR_PATH_TO_INPUT_DATA>
5161
```
5262

5363
### 3. Create a simple role for AWS Glue job execution
5464

5565
- **Create the Role**:
5666

5767
```shell
58-
aws iam create-role --role-name GlueProcessorRole --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
68+
aws iam create-role \
69+
--role-name GlueProcessorRole \
70+
--assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "glue.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
5971
```
6072

6173
- **Attach Policies to the Role**:
6274

6375
Here we grant full access to S3 as an example. For production environments it's important to restrict access appropriately.
6476
6577
```shell
66-
aws iam attach-role-policy --role-name GlueProcessorRole --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
67-
aws iam attach-role-policy --role-name GlueProcessorRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
78+
aws iam attach-role-policy \
79+
--role-name GlueProcessorRole \
80+
--policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
81+
aws iam attach-role-policy \
82+
--role-name GlueProcessorRole \
83+
--policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
6884
```
6985
7086
### 4. Create and run the job
@@ -74,15 +90,22 @@ First things first, AWS Glue jobs run a single python script, but you can includ
7490
Ensure all paths are correctly replaced with the actual ones:
7591
7692
```shell
77-
aws glue create-job --name test_hamilton_script --role GlueProcessorRole --command '{"Name" : "pythonshell", "PythonVersion": "3.9", "ScriptLocation" : "s3://<YOUR_PATH_TO_SCRIPT>/processing.py"}' --max-capacity 0.0625 --default-arguments '{"--extra-py-files" : "s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl", "--additional-python-modules" : "sf-hamilton"}'
93+
aws glue create-job \
94+
--name test_hamilton_script \
95+
--role GlueProcessorRole \
96+
--command '{"Name" : "pythonshell", "PythonVersion": "3.9", "ScriptLocation" : "s3://<YOUR_PATH_TO_SCRIPT>/processing.py"}' \
97+
--max-capacity 0.0625 \
98+
--default-arguments '{"--extra-py-files" : "s3://<YOUR_PATH_TO_WHL>/hamilton_functions-0.1-py3-none-any.whl", "--additional-python-modules" : "sf-hamilton"}'
7899
```
79100
80101
- **Run the job:**
81102
82103
Ensure all paths are correctly replaced with the actual ones:
83104
84105
```shell
85-
aws glue start-job-run --job-name test_hamilton_script --arguments '{"--input-table" : "s3://<YOUR_PATH_TO_INPUT_DATA>", "--output-table" : "s3://<YOUR_PATH_TO_OUTPUT_DATA>"}'
106+
aws glue start-job-run \
107+
--job-name test_hamilton_script \
108+
--arguments '{"--input-table" : "s3://<YOUR_PATH_TO_INPUT_DATA>", "--output-table" : "s3://<YOUR_PATH_TO_OUTPUT_DATA>"}'
86109
```
87110
88111
Once you've run the job, you should see an output file at `s3://<YOUR_PATH_TO_OUTPUT_DATA>`.

examples/aws/lambda/README.md

+61-32
Original file line numberDiff line numberDiff line change
@@ -5,63 +5,92 @@
55
Here we have an example how to deploy "hello-world" AWS Lambda with Hamilton functions.
66
This example is based on the official instruction: https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-instructions
77

8-
0. Set up AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html
8+
## Prerequisites
99

10-
1. Docker image build:
10+
- **AWS CLI Setup**: Make sure the AWS CLI is set up on your machine. If you haven't done this yet, no worries! You can follow the [Quick Start guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) for easy setup instructions.
1111

12-
```shell
13-
docker build --platform linux/amd64 -t aws-lambda-hamilton .
14-
```
12+
## Step-by-Step Guide
1513

16-
2. Local tests:
14+
### 1. Build Docker image:
1715

18-
```shell
19-
docker run -p 9000:8080 aws-lambda-hamilton
20-
```
16+
- **Build Docker image for deploy in AWS ECR**
2117

22-
```shell
23-
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": {"columns":["signups","spend"],"index":[0,1,2,3,4,5],"data":[[1,10],[10,10],[50,20],[100,40],[200,40],[400,50]]}}'
24-
```
18+
```shell
19+
docker build --platform linux/amd64 -t aws-lambda-hamilton .
20+
```
2521

26-
3. Create AWS ECR repository:
22+
- **Local tests:**
2723

24+
Run Docker container:
2825

29-
```shell
30-
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
31-
```
26+
```shell
27+
docker run -p 9000:8080 aws-lambda-hamilton
28+
```
3229

33-
```shell
34-
aws ecr create-repository --repository-name aws-lambda-hamilton --region us-east-1 --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE
35-
```
30+
Send test request to check if Docker container executes it correctly:
3631

37-
4. Deploy image to AWS ECR:
32+
```shell
33+
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": {"columns":["signups","spend"],"index":[0,1,2,3,4,5],"data":[[1,10],[10,10],[50,20],[100,40],[200,40],[400,50]]}}'
34+
```
3835

39-
```shell
40-
docker tag aws-lambda-hamilton 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-lambda-hamilton:latest
41-
```
36+
### 2. Create AWS ECR repository:
37+
38+
Ensure the AWS account number (`111122223333`) is correctly replaced with yours:
39+
40+
- **Authenticate Docker to Amazon ECR**:
41+
42+
Retrieve an authentication token to authenticate your Docker client to your Amazon Elastic Container Registry (ECR):
43+
44+
```shell
45+
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
46+
```
47+
48+
- **Create the ECR Repository**:
49+
50+
```shell
51+
aws ecr create-repository --repository-name aws-lambda-hamilton \
52+
--region us-east-1 \
53+
--image-scanning-configuration scanOnPush=true \
54+
--image-tag-mutability MUTABLE
55+
```
56+
57+
### 3. Deploy the Image to AWS ECR
58+
59+
Ensure the AWS account number (`111122223333`) is correctly replaced with yours:
4260

4361
```shell
62+
docker tag aws-lambda-hamilton 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-lambda-hamilton:latest
4463
docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-lambda-hamilton:latest
4564
```
4665

47-
4.5. Create simple AWS Lambda role (if needed):
66+
### 4. Create a simple AWS Lambda role:
67+
68+
Example of creating an AWS Role for Lambda execution:
4869

4970
```shell
50-
aws iam create-role --role-name lambda-ex --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
71+
aws iam create-role \
72+
--role-name lambda-ex \
73+
--assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
5174
```
5275

53-
5. Create AWS Lambda
76+
### 5. Create AWS Lambda
77+
78+
Ensure the AWS account number (`111122223333`) is correctly replaced with yours:
5479

5580
```shell
5681
aws lambda create-function \
57-
--function-name aws-lambda-hamilton \
58-
--package-type Image \
59-
--code ImageUri=111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-lambda-hamilton:latest \
60-
--role arn:aws:iam::111122223333:role/lambda-ex
82+
--function-name aws-lambda-hamilton \
83+
--package-type Image \
84+
--code ImageUri=111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-lambda-hamilton:latest \
85+
--role arn:aws:iam::111122223333:role/lambda-ex
6186
```
6287

63-
6. Test AWS Lambda
88+
### 6. Test AWS Lambda
6489

6590
```shell
66-
aws lambda invoke --function-name aws-lambda-hamilton --cli-binary-format raw-in-base64-out --payload '{"body": {"columns":["signups","spend"],"index":[0,1,2,3,4,5],"data":[[1,10],[10,10],[50,20],[100,40],[200,40],[400,50]]}}' response.json
91+
aws lambda invoke \
92+
--function-name aws-lambda-hamilton \
93+
--cli-binary-format raw-in-base64-out \
94+
--payload '{"body": {"columns":["signups","spend"],"index":[0,1,2,3,4,5],"data":[[1,10],[10,10],[50,20],[100,40],[200,40],[400,50]]}}' \
95+
response.json
6796
```

examples/aws/sagemaker/README.md

+23-9
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Deploying Hamilton Functions as an AWS SageMaker Processing Job
1+
# Run Hamilton Functions as an AWS SageMaker Processing Job
22

33
[AWS SageMaker](https://aws.amazon.com/sagemaker/) is a comprehensive platform that facilitates the creation, training, and deployment of machine learning (ML) models. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html) using Hamilton functions on SageMaker.
44

@@ -13,12 +13,14 @@
1313
Navigate to the container directory and build the Docker image:
1414

1515
```shell
16-
cd container/ && docker build --platform linux/amd64 -t aws-sagemaker-hamilton . && cd ..
16+
cd container/ \
17+
&& docker build --platform linux/amd64 -t aws-sagemaker-hamilton . \
18+
&& cd ..
1719
```
1820

1921
### 2. Create AWS ECR repository.
2022

21-
Replace `111122223333` with your AWS account number.
23+
Ensure the AWS account number (`111122223333`) is correctly replaced with yours:
2224

2325
- **Authenticate Docker to Amazon ECR**:
2426

@@ -31,12 +33,16 @@ Replace `111122223333` with your AWS account number.
3133
- **Create the ECR Repository**:
3234

3335
```shell
34-
aws ecr create-repository --repository-name aws-sagemaker-hamilton --region us-east-1 --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE
36+
aws ecr create-repository \
37+
--repository-name aws-sagemaker-hamilton \
38+
--region us-east-1 \
39+
--image-scanning-configuration scanOnPush=true \
40+
--image-tag-mutability MUTABLE
3541
```
3642

3743
### 3. Deploy the Image to AWS ECR
3844

39-
Ensure the AWS account number is correctly replaced with yours:
45+
Ensure the AWS account number (`111122223333`) is correctly replaced with yours:
4046

4147
```shell
4248
docker tag aws-sagemaker-hamilton 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-sagemaker-hamilton:latest
@@ -50,17 +56,25 @@ docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/aws-sagemaker-hamilton:
5056
Example of creating an AWS Role with full permissions for ECR and S3.
5157

5258
```shell
53-
aws iam create-role --role-name SageMakerScriptProcessorRole --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
59+
aws iam create-role \
60+
--role-name SageMakerScriptProcessorRole \
61+
--assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
5462
```
5563

5664
- **Attach Policies to the Role**:
5765

5866
Here we grant full access to ECR, S3 and SageMaker as an example. For production environments it's important to restrict access appropriately.
5967
6068
```shell
61-
aws iam attach-role-policy --role-name SageMakerScriptProcessorRole --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
62-
aws iam attach-role-policy --role-name SageMakerScriptProcessorRole --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
63-
aws iam attach-role-policy --role-name SageMakerScriptProcessorRole --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
69+
aws iam attach-role-policy \
70+
--role-name SageMakerScriptProcessorRole \
71+
--policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
72+
aws iam attach-role-policy \
73+
--role-name SageMakerScriptProcessorRole \
74+
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
75+
aws iam attach-role-policy \
76+
--role-name SageMakerScriptProcessorRole \
77+
--policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
6478
```
6579
6680
### 5. Install additional requirements

0 commit comments

Comments
 (0)