Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Using Ingress Controller with SCF

Jump to bottom
Thulio Ferraz Assis edited this page Apr 25, 2019 · 14 revisions

This page summarizes the steps for deploying SCF with an Ingress Controller (note that only nginx Ingress Controller has been tested). Deployment on public cloud platforms will be similar where the Ingress Controller is set up as a Kube service of type LoadBalancer whereas on Vagrant and CaaSP we'll need to make use of the controller.service.externalIPs values parameter to point to the master(?) node.

Background

On a public cloud platform SCF services can be deployed as load balanced services (services.loadbalanced) which sets up one or more (usually L4) cloud provider load balancers for the external services (1 in UAA, 3 in SCF namespace). We also need to keep track of different ports the services are exposed on (2793 for UAA, (4)443 for GoRouter etc.). The benefit of using an Ingress Controller is there is a) only one load balancer, b) SSL can be terminated on the controller and c) all traffic can go through the 80/443 ports on the controller (the Ingress routing rules manage the traffic flow to the appropriate backend services).

Steps

  • Set up your k8s cluster in preparation for deploying SCF. This includes installing Helm and the other pre-requisites around storage classes etc. Please follow the instructions from the documentation.

  • Install the nginx Ingress Controller

    helm install --name nginx-ingress --namespace ingress suse/nginx-ingress --set rbac.create=true

    After a while an Ingress Controller service will be deployed with an external IP/hostname (IP on AKS/GKE, hostname on EKS)

    kubectl get svc nginx-ingress-controller -n ingress

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      
nginx-ingress-controller   LoadBalancer   10.63.248.70   35.233.191.177   80:30344/TCP,443:31386/TCP
  • Set up appropriate DNS records (CNAME for EKS, A records for AKS and GKE) corresponding to the controller service hostname/IP with the following entries (replace yourdomain.com with the value of the actual domain):

yourdomain.com *.yourdomain.com uaa.yourdomain.com *.uaa.yourdomain.com

  • Make the following changes/additions in the Helm chart values.yaml to trigger the creation of the Ingress objects:
UAA_PORT: 443
...
ingress:
  enabled: true
  tls:
    crt: |
      <crt in PEM format>
    key: |
      <key in PEM format>

Note the port changes to make sure all communications to UAA are routed through the Ingress Controller.

  • Deploy UAA through Helm.

helm install --namespace uaa <UAA chart location on the filesystem> --name <name> --values <values yaml>

It may be helpful to follow the Ingress Controller logs (kubectl logs <IC-pod> -f) during deployment as you'll be able to spot any certificate related errors, e.g., if the certificate and/or key specified in the Ingress definition is invalid the Ingress Controller will fall back on using its own 'fake certificate'. If that happens, fix the errors and redeploy UAA.

  • Once all UAA pods are up and ready, check if UAA is working by running:

curl --cacert <CA cert used to sign the Ingress Controller cert> https://uaa.yourdomain.com/.well-known/openid-configuration should return the expected data without any certificate validation errors.

  • Update the Ingress Controller to set up TCP forwarding (note this works only on the nginx Ingress Controller):
helm upgrade nginx-ingress suse/nginx-ingress \
  --reuse-values \
  --set "tcp.20000=scf/tcp-router-tcp-router-public:20000" \
  --set "tcp.20001=scf/tcp-router-tcp-router-public:20001" \
  --set "tcp.20002=scf/tcp-router-tcp-router-public:20002" \
  --set "tcp.20003=scf/tcp-router-tcp-router-public:20003" \
  --set "tcp.20004=scf/tcp-router-tcp-router-public:20004" \
  --set "tcp.20005=scf/tcp-router-tcp-router-public:20005" \
  --set "tcp.20006=scf/tcp-router-tcp-router-public:20006" \
  --set "tcp.20007=scf/tcp-router-tcp-router-public:20007" \
  --set "tcp.20008=scf/tcp-router-tcp-router-public:20008" \
  --set "tcp.2222=scf/diego-ssh-ssh-proxy-public:2222"
  • Set up the SCF deployment to trust the CA certificate that signed the certificate on the Ingress Controller.

export INGRESS_CA_CERT=$(cat your-ingress-cacert.pem)

  • Make the changes/additions to values.yaml similarly to those made for UAA. Use the TLS cert and key for the bare DOMAIN and *.DOMAIN here, as opposed to the UAA above.
UAA_PORT: 443
UAA_PUBLIC_PORT: 443
...
ingress:
  enabled: true
  tls:
    crt: |
      <crt in PEM format>
    key: |
      <key in PEM format>
  • Deploy SCF.

helm install --namespace scf <SCF chart location> --name <name> --set "secrets.UAA_CA_CERT=${INGRESS_CA_CERT} --values <values yaml> etc.

  • Once the SCF pods are all up and ready (2/2 and 1/1 with the exception of the secrets-generator and post-deployment pods which should both have completed) you can login via the CF CLI.

cf login https://api.yourdomain.com -u <user> -p <passwd>

Clone this wiki locally