Skip to content

Commit

Permalink
Merge pull request #93 from awslabs/links
Browse files Browse the repository at this point in the history
Update repository URLs from aws-controllers-k8s to awslabs
  • Loading branch information
a-hilaly authored Nov 8, 2024
2 parents 19a68f0 + eeeacfb commit d811ba5
Show file tree
Hide file tree
Showing 6 changed files with 162 additions and 90 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -177,13 +177,13 @@ $(CONTROLLER_GEN): $(LOCALBIN)
.PHONY: image
build-image: ## Build the Symphony controller images using ko build
$(WITH_GOFLAGS) KOCACHE=$(KOCACHE) KO_DOCKER_REPO="095708837592.dkr.ecr.us-west-2.amazonaws.com/symphony" \
ko build --bare github.com/aws-controllers-k8s/symphony/cmd/controller \
ko build --bare github.com/awslabs/symphony/cmd/controller \
--push=false --tags ${RELEASE_VERSION} --sbom=none

.PHONY: publish
publish-image: ## Publish the Symphony controller images to ECR
$(WITH_GOFLAGS) KOCACHE=$(KOCACHE) KO_DOCKER_REPO="095708837592.dkr.ecr.us-west-2.amazonaws.com/symphony" \
ko publish --bare github.com/aws-controllers-k8s/symphony/cmd/controller \
ko publish --bare github.com/awslabs/symphony/cmd/controller \
--tags ${RELEASE_VERSION} --sbom=none

.PHONY: package-helm
Expand Down
138 changes: 98 additions & 40 deletions examples/eks-cluster-mgmt/README.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,53 @@
# Amazon EKS cluster management using Symphony & ACK
This example demonstrates how to manage a fleet of EKS clusters using Symphony, ACK, and ArgoCD -- it creates EKS clusters, and bootstraps them with the required add-ons

A hub-spoke model is used in this example; a management cluster (hub) is created as part of the initial setup and the controllers needed for provisioning and bootstrapping workload clusters (spokes) are installed on top.
This example demonstrates how to manage a fleet of EKS clusters using Symphony,
ACK, and ArgoCD -- it creates EKS clusters, and bootstraps them with the
required add-ons

A hub-spoke model is used in this example; a management cluster (hub) is created
as part of the initial setup and the controllers needed for provisioning and
bootstrapping workload clusters (spokes) are installed on top.

**NOTE:** As this example evolves, some of the instructions below will be detailed further (e.g. the creation of the management cluster), others (e.g. controllers installation) will be automated via the GitOps flow.
**NOTE:** As this example evolves, some of the instructions below will be
detailed further (e.g. the creation of the management cluster), others (e.g.
controllers installation) will be automated via the GitOps flow.

## Prerequisites

1. AWS account for the management cluster
2. AWS account for workload clusters; each with the following IAM roles:
- `eks-cluster-mgmt-ec2`
- `eks-cluster-mgmt-eks`
- `eks-cluster-mgmt-iam`

The permissions should be as needed for every controller.
Trust policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<mgmt-account-id>:role/ack-<srvc-name>-controller"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
```

- `eks-cluster-mgmt-ec2`
- `eks-cluster-mgmt-eks`
- `eks-cluster-mgmt-iam`

The permissions should be as needed for every controller. Trust policy:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<mgmt-account-id>:role/ack-<srvc-name>-controller"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
```

## Instructions

### Environment variables

1. Use the snippet below to set environment variables. Replace the placeholders first (surrounded with`<>`):
1. Use the snippet below to set environment variables. Replace the placeholders
first (surrounded with`<>`):

```sh
export SYMPHONY_REPO_URL="https://github.com/aws-controllers-k8s/private-symphony.git"
export SYMPHONY_REPO_URL="https://github.com/awslabs/private-symphony.git"
export WORKSPACE_PATH=<workspace-path> #the directory where repos will be cloned e.g. ~/environment/
export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
export AWS_REGION=<region> #e.g. us-west-2
Expand All @@ -45,44 +56,60 @@ export ARGOCD_CHART_VERSION=7.5.2
```

### Management cluster

2. Create an EKS cluster (management cluster)
3. Create IAM OIDC provider for the cluster:

```sh
eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --approve
```

4. Save OIDC provider URL in an environment variable:

```sh
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
```

5. Install the following ACK controllers on the management cluster:
- ACK IAM controller
- ACK EC2 controller
- ACK EKS controller
6. Install Symphony on the management cluster. Please note that this example is tested on 0.1.0-rc.3.
- ACK IAM controller
- ACK EC2 controller
- ACK EKS controller
6. Install Symphony on the management cluster. Please note that this example is
tested on 0.1.0-rc.3.
7. Install EKS pod identity add-on:

```sh
aws eks create-addon --cluster-name $CLUSTER_NAME --addon-name eks-pod-identity-agent --addon-version v1.0.0-eksbuild.1
```

### Repo

8. Clone Symphony repo:

```sh
git clone $SYMPHONY_REPO_URL $WORKSPACE_PATH/symphony
```

9. Create the GitHub repo `cluster-mgmt` in your organization; it will contain the clusters definition, and it will be reconciled to the management cluster via the GitOps flow
9. Create the GitHub repo `cluster-mgmt` in your organization; it will contain
the clusters definition, and it will be reconciled to the management cluster
via the GitOps flow

**NOTE:** Until Symphony is released, make sure the repo you create is private.

10. Save the URL of the created repo in an environment variable:

```sh
export MY_REPO_URL=<repo-url> #e.g. https://github.com/iamahgoub/cluster-mgmt.git
```

11. Clone the created repo:

```sh
git clone $MY_REPO_URL $WORKSPACE_PATH/cluster-mgmt
```

12. Populate the repo:

```sh
cp -r $WORKSPACE_PATH/symphony/examples/cluster-mgmt/* $WORKSPACE_PATH/cluster-mgmt
find /path/to/directory -type f -exec sed -i "s/search_string/$REPLACE_STRING/g" {} +
Expand All @@ -93,7 +120,9 @@ find $WORKSPACE_PATH/cluster-mgmt -type f -exec sed -i "s~AWS_REGION~$AWS_REGION
find $WORKSPACE_PATH/cluster-mgmt -type f -exec sed -i "s~CLUSTER_NAME~$CLUSTER_NAME~g" {} +
find $WORKSPACE_PATH/cluster-mgmt -type f -exec sed -i "s~OIDC_PROVIDER~$OIDC_PROVIDER~g" {} +
```

13. Push the changes

```sh
cd $WORKSPACE_PATH/cluster-mgmt
git add .
Expand All @@ -103,7 +132,10 @@ cd $WORKSPACE_PATH
```

### ArgoCD installation
14. Create an IAM role for ArgoCD on the management cluster and associated with ArgoCD `ServiceAccount`:

14. Create an IAM role for ArgoCD on the management cluster and associated with
ArgoCD `ServiceAccount`:

```sh
cat >argocd-policy.json <<EOF
{
Expand Down Expand Up @@ -147,7 +179,9 @@ aws iam attach-role-policy --role-name argocd-hub-role --policy-arn=arn:aws:iam:

aws eks create-pod-identity-association --cluster-name $CLUSTER_NAME --role-arn arn:aws:iam::$ACCOUNT_ID:role/argocd-hub-role --namespace argocd --service-account argocd-application-controller
```

15. Install ArgoCD helm chart:

```sh
helm repo add argo-cd https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo-cd/argo-cd --version $ARGOCD_CHART_VERSION \
Expand All @@ -158,44 +192,68 @@ helm upgrade --install argocd argo-cd/argo-cd --version $ARGOCD_CHART_VERSION \

### Bootstrapping

16. Create ArgoCD `Repository` resource that points to `cluster-mgmt` repo created in an earlier instruction
16. Create ArgoCD `Repository` resource that points to `cluster-mgmt` repo
created in an earlier instruction
17. Apply the bootstrap ArgoCD application:

```sh
kubectl apply -f $WORKSPACE_PATH/cluster-mgmt/gitops/bootstrap.yaml
```

### Adding workload clusters

The initial configuration creates one workload cluster named `workload-cluster1`.
The initial configuration creates one workload cluster named
`workload-cluster1`.

**TODO:** add steps for cluster/account mapping

18. Add a workload cluster by adding a manifest for it under `clusters/`. Refer to `clusters/workload-cluster1.yaml` as an example.
18. Add a workload cluster by adding a manifest for it under `clusters/`. Refer
to `clusters/workload-cluster1.yaml` as an example.
19. Include the new cluster manifest in `clusters/kustomization.yaml`.
20. Add the cluster name and corresponding account number in `charts-values/ack-multi-account/values.yaml`.
20. Add the cluster name and corresponding account number in
`charts-values/ack-multi-account/values.yaml`.
21. Commit/push the changes to Git.


## Known issues
1. You will need to restart the Symphony controller when you add a new workload cluster due to a bug in the controller. Once the resource group `eksclusterwithvpc` is applied, the controller is able to apply the corresponding VPC resources, but it is not able to recognize the generated ids (e.g. subnet id), and feed that into EKS resources. Refer to [this issue](https://github.com/aws-controllers-k8s/private-symphony/issues/8) for more details.
2. Deleting a cluster does not properly clean up all cluster resources i.e. subnets, routetables are left strangling. ACK EC2 controller keep reporting dependencies preventing deletion. To work around this issue, attempt restart ACK EC2 controller, and/or manually deleting the resources.

1. You will need to restart the Symphony controller when you add a new workload
cluster due to a bug in the controller. Once the resource group
`eksclusterwithvpc` is applied, the controller is able to apply the
corresponding VPC resources, but it is not able to recognize the generated
ids (e.g. subnet id), and feed that into EKS resources. Refer to
[this issue](https://github.com/awslabs/private-symphony/issues/8) for more
details.
2. Deleting a cluster does not properly clean up all cluster resources i.e.
subnets, routetables are left strangling. ACK EC2 controller keep reporting
dependencies preventing deletion. To work around this issue, attempt restart
ACK EC2 controller, and/or manually deleting the resources.

## Clean-up
1. Delete ArgoCD bootstrap application, and wait for workload clusters and hosting VPCs to be deleted:

1. Delete ArgoCD bootstrap application, and wait for workload clusters and
hosting VPCs to be deleted:

```sh
kubectl delete application bootstrap -n argocd
```

2. Uninstall ArgoCD helm chart

```sh
helm uninstall argocd -n argocd
```

3. Delete ArgoCD IAM role and policy

```sh
aws iam delete-role --role-name argocd-hub-role
```

4. Delete ArgoCD IAM policy

```sh
aws iam delete-policy --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/argocd-policy
```

5. Delete ACK controllers and Symphony
6. Delete the management cluster
2 changes: 1 addition & 1 deletion website/docs/docs/getting-started/01-Installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Once authenticated, install Symphony using the Helm chart:
```sh
# Fetch the latest release version from GitHub
export SYMPHONY_VERSION=$(curl -s \
https://api.github.com/repos/aws-controllers-k8s/private-symphony/releases/latest | \
https://api.github.com/repos/awslabs/private-symphony/releases/latest | \
grep '"tag_name":' | \
sed -E 's/.*"([^"]+)".*/\1/' \
)
Expand Down
Loading

0 comments on commit d811ba5

Please sign in to comment.