diff --git a/.gitignore b/.gitignore index 1550bdd..29b636a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,13 +1,2 @@ -/artifacts -/target - -*Cargo.lock - .idea -*.iml - -.terraform -*.tfstate* -*.tfvars* - -*.zip \ No newline at end of file +*.iml \ No newline at end of file diff --git a/README.md b/README.md index 9e4125b..859f2a1 100644 --- a/README.md +++ b/README.md @@ -1,165 +1,8 @@ # Description -The intention of this project is to build a skeleton of VRP REST API for quick prototyping using Rust and AWS. It uses -another Rust project which implements a rich VRP solver functionality, you can find it [here](https://github.com/reinterpretcat/vrp). +This repo contains various experiments with technologies, languages, tools in connection to VRP topic. -# Overview +# Content -This repo contains two implementations of REST API: - -* `server` a traditional server approach built using [actix-web framework](https://github.com/actix/actix-web) -* `serverless` a serverless approach using AWS lambdas - -## Server approach - - -## Serverless approach - -### Architecture - -On API level, there are two public endpoints: - -- __submit problem__: responsible for submitting VRP in [pragmatic format](https://reinterpretcat.github.io/vrp/concepts/pragmatic/index.html) -- __poll solution__: provides way to get calculated VRP solution back - -Essentially, the flow can be described as follows: - -![architecture](docs/architecture.png "VRP API") - -1. User create POST request to `/problem` resource which is exposed via AWS API Gateway. -2. API Gateway triggers AWS `submit_problem` lambda function. See its source code inside `./lambdas/gateway/submit` folder. -3. If problem definition is ok, the lambda function uploads it to S3 bucket as `problem.json`. As result, user gets -`submit id` which is the name of the bucket. -4. The `trigger_solver` lambda function is triggered once `problem.json` is uploaded. Its source code is inside -`./lambdas/triggers/batch` folder. -5. This lambda triggers a AWS Batch job for solving VRP. -6. The Batch job pulls problem definition from S3 bucket, solves VRP, and uploads `solution.json` -7. Any time user can poll solution by calling `/solution` resource using GET method. This resource is also exposed -via AWS API Gateway. -8. API Gateway triggers AWS `poll_problem` lambda function. See its source code inside `./lambdas/gateway/poll` folder. -9. The lambda function downloads `state.json` file (see a note below) and, depending on its content, returns solution -or submission state. - -Notes: -- additionally, there is `state.json` file in S3 bucket. It is used to track the state of submission and, potentially, -return some information regarding execution progress -- actually, batch job can be created by submit lambda, but this would limit extensibility: in real world, there -should be an extra step to request routing matrix information. It might be expensive in terms of performance and time, -so another logic might be needed here and it is better to avoid coupling problem submission and calling the solver. - - -## Source code structure - -- __Rust code__: - - `./common`: contains shared code used by different crates in the project - - `./lambdas`: contains code for AWS lambda functions used by `serverless` approach - - `./server`: contains code for server approach - - `./solver`: contains a binary crate for solving VRP problem -- __Build & Deployment & Test__ - - `./terraform`: terraform configuration to deploy AWS resources - - `./scripts`: various scripts to build, deploy, and test (see next section for details) - - `./solver/image/Dockerfile`: a docker file for solver image - -# How to use - -## Prerequisites - -Once source code is pulled from github, make sure that you have the following prerequisites: - -- linux compatible environment -- rust - * install rust toolchain using instructions from [official site](https://www.rust-lang.org/tools/install) - * install cross compilation tool from [github](https://github.com/rust-embedded/cross) which will be used to build - code to be in AWS: - ```shell script - # see https://github.com/awslabs/aws-lambda-rust-runtime/issues/17 - cargo install --force --git https://github.com/rust-embedded/cross - ``` -- AWS - * create an account in [AWS](https://aws.amazon.com/resources/create-account) - * create Elastic Container Repository with `vrp-solver` name which will be used to publish docker image - * set environment variables: - ```shell script - export AWS_ACCESS_KEY_ID=your-access-key - export AWS_SECRET_ACCESS_KEY=your-secret-key - export AWS_DEFAULT_REGION="eu-west-1" # or another one - ``` -- Terraform - * install terraform using instructions from [official site](https://learn.hashicorp.com/terraform/getting-started/install.html) - * create `private.tfvars` file inside `terraform` directory and put the following: - ``` - vpc_subnet_ids = ["subnet-12340a1b"] - vpc_security_group_ids = ["sg-12340a2b"] - batch_container_image = "012345678901.dkr.ecr.eu-west-1.amazonaws.com/vrp-solver" - ``` - Use proper subnet and security group ids (e.g. default ones) and change `012345678901` to your account id (12 digits). - -## Build - -Use the following script to build the code: - -```shell script -./scripts/build.sh -``` - -It builds rust code and copies build artifacts into `artifacts` folder. - -## Deploy - -- build a docker image of the VRP solver and publish it in Elastic Container Repository: - -```shell script -./scripts/deploy.sh -``` - -- create AWS resources using terraform: - -```shell script - cd terraform - terraform init - terraform apply -var-file="private.tfvars" -``` - -## Test serverless - -If you decided to use different AWS region, adjust url in tests scripts according to -[AWS documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-call-api.html). -Then determine api gateway id which you can find by looking into AWS console or terraform output. - -To post the test problem, run the following command: - -```shell script -./scripts/test_submit_problem.sh api_gateway_id -``` - -Replace `api_gateway_id` with real one. - -If everything is ok, you will get json response like: - -```json -{ - "id": "ecc9c997-5063-4eb3-8099-1c3c1743b4f1" -} -``` - -You can get solution or its processing status by running: - -```shell script -./scripts/test_poll_solution.sh api_gateway_id ecc9c997-5063-4eb3-8099-1c3c1743b4f1 -``` - -Result is one of the following responses: - -- `200 OK`: solution is calculated and returned in response body -- `204 No content`: problem is accepted, but solution is not yet calculated -- `404 Not found`: no problem is found -- `409 Conflict`: an error is occurred while calculating solution. Response body contains additional information - - -# Further improvements - -- limit resources accessed by each service -- add openapi spec for API Gateway -- add authorization -- add unit tests -- ... \ No newline at end of file +* **serverless-api**: experimenting with serverless computing on AWS + Rust +* **server-k8s-api**: experimenting with kubernetes + REST server on Rust \ No newline at end of file diff --git a/server/Cargo.toml b/server-k8s-api/Cargo.toml similarity index 100% rename from server/Cargo.toml rename to server-k8s-api/Cargo.toml diff --git a/server/src/main.rs b/server-k8s-api/src/main.rs similarity index 96% rename from server/src/main.rs rename to server-k8s-api/src/main.rs index a5a3bf3..8acf7f5 100644 --- a/server/src/main.rs +++ b/server-k8s-api/src/main.rs @@ -1,3 +1,5 @@ +mod routes; + use actix_web::{get, web, App, HttpServer, Responder}; #[get("/{id}/{name}/index.html")] diff --git a/server-k8s-api/src/models.rs b/server-k8s-api/src/models.rs new file mode 100644 index 0000000..e69de29 diff --git a/server-k8s-api/src/routes.rs b/server-k8s-api/src/routes.rs new file mode 100644 index 0000000..1c8fcbe --- /dev/null +++ b/server-k8s-api/src/routes.rs @@ -0,0 +1,15 @@ +use actix_web::{web, HttpResponse}; + +pub fn configure(cfg: &mut web::ServiceConfig) { + cfg + .service(web::resource("/problems").route(web::post().to_async(submit_problem))) + .service(web::resource("/problems/{id}/solutions").route(web::get().to_async(get_solutions))); +} + +async fn submit_problem(path: web::Path) -> HttpResponse { + +} + +async fn get_solutions(path: web::Path) -> HttpResponse { + +} \ No newline at end of file diff --git a/serverless-api/.gitignore b/serverless-api/.gitignore new file mode 100644 index 0000000..2a9ca82 --- /dev/null +++ b/serverless-api/.gitignore @@ -0,0 +1,10 @@ +/artifacts +/target + +*Cargo.lock + +.terraform +*.tfstate* +*.tfvars* + +*.zip \ No newline at end of file diff --git a/Cargo.toml b/serverless-api/Cargo.toml similarity index 89% rename from Cargo.toml rename to serverless-api/Cargo.toml index 40be960..3cfdf4e 100644 --- a/Cargo.toml +++ b/serverless-api/Cargo.toml @@ -3,7 +3,6 @@ members = [ "common", "lambdas", - "server", "solver", ] diff --git a/serverless-api/README.md b/serverless-api/README.md new file mode 100644 index 0000000..821f55a --- /dev/null +++ b/serverless-api/README.md @@ -0,0 +1,152 @@ +# Description + +The intention of this project is to build a skeleton of VRP REST API for quick prototyping using Rust and AWS. It uses +another Rust project which implements a rich VRP solver functionality, you can find it [here](https://github.com/reinterpretcat/vrp). + +# Architecture + +On API level, there are two public endpoints: + +- __submit problem__: responsible for submitting VRP in [pragmatic format](https://reinterpretcat.github.io/vrp/concepts/pragmatic/index.html) +- __poll solution__: provides way to get calculated VRP solution back + +Essentially, the flow can be described as follows: + +![architecture](docs/architecture.png "VRP API") + +1. User create POST request to `/problem` resource which is exposed via AWS API Gateway. +2. API Gateway triggers AWS `submit_problem` lambda function. See its source code inside `./lambdas/gateway/submit` folder. +3. If problem definition is ok, the lambda function uploads it to S3 bucket as `problem.json`. As result, user gets +`submit id` which is the name of the bucket. +4. The `trigger_solver` lambda function is triggered once `problem.json` is uploaded. Its source code is inside +`./lambdas/triggers/batch` folder. +5. This lambda triggers a AWS Batch job for solving VRP. +6. The Batch job pulls problem definition from S3 bucket, solves VRP, and uploads `solution.json` +7. Any time user can poll solution by calling `/solution` resource using GET method. This resource is also exposed +via AWS API Gateway. +8. API Gateway triggers AWS `poll_problem` lambda function. See its source code inside `./lambdas/gateway/poll` folder. +9. The lambda function downloads `state.json` file (see a note below) and, depending on its content, returns solution +or submission state. + +Notes: +- additionally, there is `state.json` file in S3 bucket. It is used to track the state of submission and, potentially, +return some information regarding execution progress +- actually, batch job can be created by submit lambda, but this would limit extensibility: in real world, there +should be an extra step to request routing matrix information. It might be expensive in terms of performance and time, +so another logic might be needed here and it is better to avoid coupling problem submission and calling the solver. + + +# Source code structure + +- __Rust code__: + - `./common`: contains shared code used by different crates in the project + - `./lambdas`: contains code for AWS lambda functions + - `./solver`: contains a binary crate for solving VRP problem +- __Build & Deployment & Test__ + - `./terraform`: terraform configuration to deploy AWS resources + - `./scripts`: various scripts to build, deploy, and test (see next section for details) + - `./solver/image/Dockerfile`: a docker file for solver image + +# How to use + +## Prerequisites + +Once source code is pulled from github, make sure that you have the following prerequisites: + +- linux compatible environment +- rust + * install rust toolchain using instructions from [official site](https://www.rust-lang.org/tools/install) + * install cross compilation tool from [github](https://github.com/rust-embedded/cross) which will be used to build + code to be in AWS: + ```shell script + # see https://github.com/awslabs/aws-lambda-rust-runtime/issues/17 + cargo install --force --git https://github.com/rust-embedded/cross + ``` +- AWS + * create an account in [AWS](https://aws.amazon.com/resources/create-account) + * create Elastic Container Repository with `vrp-solver` name which will be used to publish docker image + * set environment variables: + ```shell script + export AWS_ACCESS_KEY_ID=your-access-key + export AWS_SECRET_ACCESS_KEY=your-secret-key + export AWS_DEFAULT_REGION="eu-west-1" # or another one + ``` +- Terraform + * install terraform using instructions from [official site](https://learn.hashicorp.com/terraform/getting-started/install.html) + * create `private.tfvars` file inside `terraform` directory and put the following: + ``` + vpc_subnet_ids = ["subnet-12340a1b"] + vpc_security_group_ids = ["sg-12340a2b"] + batch_container_image = "012345678901.dkr.ecr.eu-west-1.amazonaws.com/vrp-solver" + ``` + Use proper subnet and security group ids (e.g. default ones) and change `012345678901` to your account id (12 digits). + +## Build + +Use the following script to build the code: + +```shell script +./scripts/build.sh +``` + +It builds rust code and copies build artifacts into `artifacts` folder. + +## Deploy + +- build a docker image of the VRP solver and publish it in Elastic Container Repository: + +```shell script +./scripts/deploy.sh +``` + +- create AWS resources using terraform: + +```shell script + cd terraform + terraform init + terraform apply -var-file="private.tfvars" +``` + +## Test + +If you decided to use different AWS region, adjust url in tests scripts according to +[AWS documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-call-api.html). +Then determine api gateway id which you can find by looking into AWS console or terraform output. + +To post the test problem, run the following command: + +```shell script +./scripts/test_submit_problem.sh api_gateway_id +``` + +Replace `api_gateway_id` with real one. + +If everything is ok, you will get json response like: + +```json +{ + "id": "ecc9c997-5063-4eb3-8099-1c3c1743b4f1" +} +``` + +You can get solution or its processing status by running: + +```shell script +./scripts/test_poll_solution.sh api_gateway_id ecc9c997-5063-4eb3-8099-1c3c1743b4f1 +``` + +Result is one of the following responses: + +- `200 OK`: solution is calculated and returned in response body +- `204 No content`: problem is accepted, but solution is not yet calculated +- `404 Not found`: no problem is found +- `409 Conflict`: an error is occurred while calculating solution. Response body contains additional information + + +# Further improvements + +- limit resources accessed by each service +- add openapi spec for API Gateway +- add authorization +- add unit tests +- ... \ No newline at end of file diff --git a/common/Cargo.toml b/serverless-api/common/Cargo.toml similarity index 100% rename from common/Cargo.toml rename to serverless-api/common/Cargo.toml diff --git a/common/src/aws/mod.rs b/serverless-api/common/src/aws/mod.rs similarity index 100% rename from common/src/aws/mod.rs rename to serverless-api/common/src/aws/mod.rs diff --git a/common/src/aws/s3_ops.rs b/serverless-api/common/src/aws/s3_ops.rs similarity index 100% rename from common/src/aws/s3_ops.rs rename to serverless-api/common/src/aws/s3_ops.rs diff --git a/common/src/lib.rs b/serverless-api/common/src/lib.rs similarity index 100% rename from common/src/lib.rs rename to serverless-api/common/src/lib.rs diff --git a/common/src/models/errors.rs b/serverless-api/common/src/models/errors.rs similarity index 100% rename from common/src/models/errors.rs rename to serverless-api/common/src/models/errors.rs diff --git a/common/src/models/mod.rs b/serverless-api/common/src/models/mod.rs similarity index 100% rename from common/src/models/mod.rs rename to serverless-api/common/src/models/mod.rs diff --git a/common/src/models/states.rs b/serverless-api/common/src/models/states.rs similarity index 100% rename from common/src/models/states.rs rename to serverless-api/common/src/models/states.rs diff --git a/common/src/runtime/environment.rs b/serverless-api/common/src/runtime/environment.rs similarity index 100% rename from common/src/runtime/environment.rs rename to serverless-api/common/src/runtime/environment.rs diff --git a/common/src/runtime/mod.rs b/serverless-api/common/src/runtime/mod.rs similarity index 100% rename from common/src/runtime/mod.rs rename to serverless-api/common/src/runtime/mod.rs diff --git a/common/src/runtime/variables.rs b/serverless-api/common/src/runtime/variables.rs similarity index 100% rename from common/src/runtime/variables.rs rename to serverless-api/common/src/runtime/variables.rs diff --git a/common/tests/unit/models/error_test.rs b/serverless-api/common/tests/unit/models/error_test.rs similarity index 100% rename from common/tests/unit/models/error_test.rs rename to serverless-api/common/tests/unit/models/error_test.rs diff --git a/docs/architecture.png b/serverless-api/docs/architecture.png similarity index 100% rename from docs/architecture.png rename to serverless-api/docs/architecture.png diff --git a/docs/architecture.svg b/serverless-api/docs/architecture.svg similarity index 100% rename from docs/architecture.svg rename to serverless-api/docs/architecture.svg diff --git a/lambdas/Cargo.toml b/serverless-api/lambdas/Cargo.toml similarity index 100% rename from lambdas/Cargo.toml rename to serverless-api/lambdas/Cargo.toml diff --git a/lambdas/src/common/batch_ops.rs b/serverless-api/lambdas/src/common/batch_ops.rs similarity index 100% rename from lambdas/src/common/batch_ops.rs rename to serverless-api/lambdas/src/common/batch_ops.rs diff --git a/lambdas/src/common/mod.rs b/serverless-api/lambdas/src/common/mod.rs similarity index 100% rename from lambdas/src/common/mod.rs rename to serverless-api/lambdas/src/common/mod.rs diff --git a/lambdas/src/common/proxy_events.rs b/serverless-api/lambdas/src/common/proxy_events.rs similarity index 100% rename from lambdas/src/common/proxy_events.rs rename to serverless-api/lambdas/src/common/proxy_events.rs diff --git a/lambdas/src/gateway/poll/main.rs b/serverless-api/lambdas/src/gateway/poll/main.rs similarity index 100% rename from lambdas/src/gateway/poll/main.rs rename to serverless-api/lambdas/src/gateway/poll/main.rs diff --git a/lambdas/src/gateway/submit/main.rs b/serverless-api/lambdas/src/gateway/submit/main.rs similarity index 100% rename from lambdas/src/gateway/submit/main.rs rename to serverless-api/lambdas/src/gateway/submit/main.rs diff --git a/lambdas/src/lib.rs b/serverless-api/lambdas/src/lib.rs similarity index 100% rename from lambdas/src/lib.rs rename to serverless-api/lambdas/src/lib.rs diff --git a/lambdas/src/triggers/batch/main.rs b/serverless-api/lambdas/src/triggers/batch/main.rs similarity index 100% rename from lambdas/src/triggers/batch/main.rs rename to serverless-api/lambdas/src/triggers/batch/main.rs diff --git a/lambdas/tests/unit/common/proxy_events_test.rs b/serverless-api/lambdas/tests/unit/common/proxy_events_test.rs similarity index 100% rename from lambdas/tests/unit/common/proxy_events_test.rs rename to serverless-api/lambdas/tests/unit/common/proxy_events_test.rs diff --git a/scripts/build.sh b/serverless-api/scripts/build.sh similarity index 100% rename from scripts/build.sh rename to serverless-api/scripts/build.sh diff --git a/scripts/deploy.sh b/serverless-api/scripts/deploy.sh similarity index 100% rename from scripts/deploy.sh rename to serverless-api/scripts/deploy.sh diff --git a/scripts/test_data.json b/serverless-api/scripts/test_data.json similarity index 100% rename from scripts/test_data.json rename to serverless-api/scripts/test_data.json diff --git a/scripts/test_poll_solution.sh b/serverless-api/scripts/test_poll_solution.sh similarity index 100% rename from scripts/test_poll_solution.sh rename to serverless-api/scripts/test_poll_solution.sh diff --git a/scripts/test_submit_problem.sh b/serverless-api/scripts/test_submit_problem.sh similarity index 100% rename from scripts/test_submit_problem.sh rename to serverless-api/scripts/test_submit_problem.sh diff --git a/solver/Cargo.toml b/serverless-api/solver/Cargo.toml similarity index 100% rename from solver/Cargo.toml rename to serverless-api/solver/Cargo.toml diff --git a/solver/image/Dockerfile b/serverless-api/solver/image/Dockerfile similarity index 100% rename from solver/image/Dockerfile rename to serverless-api/solver/image/Dockerfile diff --git a/solver/src/main.rs b/serverless-api/solver/src/main.rs similarity index 100% rename from solver/src/main.rs rename to serverless-api/solver/src/main.rs diff --git a/terraform/api.tf b/serverless-api/terraform/api.tf similarity index 100% rename from terraform/api.tf rename to serverless-api/terraform/api.tf diff --git a/terraform/batch.tf b/serverless-api/terraform/batch.tf similarity index 100% rename from terraform/batch.tf rename to serverless-api/terraform/batch.tf diff --git a/terraform/iam.tf b/serverless-api/terraform/iam.tf similarity index 100% rename from terraform/iam.tf rename to serverless-api/terraform/iam.tf diff --git a/terraform/lambda.tf b/serverless-api/terraform/lambda.tf similarity index 100% rename from terraform/lambda.tf rename to serverless-api/terraform/lambda.tf diff --git a/terraform/main.tf b/serverless-api/terraform/main.tf similarity index 100% rename from terraform/main.tf rename to serverless-api/terraform/main.tf diff --git a/terraform/storage.tf b/serverless-api/terraform/storage.tf similarity index 100% rename from terraform/storage.tf rename to serverless-api/terraform/storage.tf diff --git a/terraform/variables.tf b/serverless-api/terraform/variables.tf similarity index 100% rename from terraform/variables.tf rename to serverless-api/terraform/variables.tf