| Feature | Status | API Version | Example | Description |
|---|---|---|---|---|
| Managed Disks | Beta | vlabs |
kubernetes-vmas.json | Description |
| Calico Network Policy | Alpha | vlabs |
kubernetes-calico.json | Description |
| Custom VNET | Beta | vlabs |
kubernetesvnet-azure-cni.json | Description |
Enabling Managed Identity configures acs-engine to include and use MSI identities for all interactions with the Azure Resource Manager (ARM) API.
Instead of using a static servic principal written to /etc/kubernetes/azure.json, Kubernetes will use a dynamic, time-limited token fetched from the MSI extension running on master and agent nodes. This support is currently alpha and requires Kubernetes v1.9.1 or newer.
Enable Managed Identity by adding useManagedIdentity in kubernetesConfig.
"kubernetesConfig": {
"useManagedIdentity": true,
"customHyperkubeImage": "docker.io/colemickens/hyperkube-amd64:3b15e8a446fa09d68a2056e2a5e650c90ae849ed"
}By default, the cluster will be provisioned with Role-Based Access Control enabled. Disable RBAC by adding enableRbac in kubernetesConfig in the api model:
"kubernetesConfig": {
"enableRbac": false
}See cluster definition for further detail.
Managed disks are supported for both node OS disks and Kubernetes persistent volumes.
Related upstream PR for details.
By default, each ACS-Engine cluster is bootstrapped with several StorageClass resources. This bootstrapping is handled by the addon-manager pod that creates resources defined under /etc/kubernetes/addons directory on master VMs.
The default storage class has been set via the Kubernetes admission controller DefaultStorageClass.
The default storage class will be used if persistent volume resources don't specify a storage class as part of the resource definition.
The default storage class uses non-managed blob storage and will provision the blob within an existing storage account present in the resource group or provision a new storage account.
Non-managed persistent volume types are available on all VM sizes.
As part of cluster bootstrapping, two storage classes will be created to provide access to create Kubernetes persistent volumes using Azure managed disks.
Nodes will be labelled as follows if they support managed disks:
storageprofile=managed
storagetier=<Standard_LRS|Premium_LRS>
They are managed-premium and managed-standard and map to Standard_LRS and Premium_LRS managed disk types respectively.
In order to use these storage classes the following conditions must be met.
- The cluster must be running Kubernetes release 1.7 or greater. Refer to this example for how to provision a Kubernetes cluster of a specific version.
- The node must support managed disks. See this example to provision nodes with managed disks. You can also confirm if a node has managed disks using kubectl.
kubectl get nodes -l storageprofile=managed
NAME STATUS AGE VERSION
k8s-agent1-23731866-0 Ready 24m v1.7.2-
The VM size must support the type of managed disk type requested. For example, Premium VM sizes with managed OS disks support both managed-standard and managed-premium storage classes whereas Standard VM sizes with managed OS disks only support managed-standard storage class.
-
If you have mixed node cluster (both non-managed and managed disk types). You must use affinity or nodeSelectors on your resource definitions in order to ensure that workloads are scheduled to VMs that support the underlying disk requirements.
For example
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: storageprofile
operator: In
values:
- managed
Kubernetes clusters are configured by default to use the Azure CNI plugin which provides an Azure native networking experience. Pods will receive IP addresses directly from the vnet subnet on which they're hosted. If the api model doesn't specify explicitly, acs-engine will automatically provide the following networkPolicy configuration in kubernetesConfig:
"kubernetesConfig": {
"networkPolicy": "azure"
}
In addition you can modify the following settings to change the networking behavior when using Azure integrated networking:
IP addresses are pre-allocated in the subnet. Using ipAddressCount you can specify how many you would like to pre-allocate. This number needs to account for number of pods you would like to run on that subnet.
"masterProfile": {
"ipAddressCount": 200
},
Currently, the IP addresses that are pre-allocated aren't allowed by the default natter for Internet bound traffic. In order to work around this limitation we allow the user to specify the vnetCidr (eg. 10.0.0.0/8) to be EXCLUDED from the default masquerade rule that is applied. The result is that traffic destined for anything within that block will NOT be natted on the outbound VM interface. This field has been called vnetCidr but may be wider than the vnet cidr block if you would like POD IPs to be routable across vnets using vnet-peering or express-route.
"masterProfile": {
"vnetCidr": "10.0.0.0/8",
},
When using Azure integrated networking the maxPods setting will be set to 30 by default. This number can be changed keeping in mind that there is a limit of 4,000 IPs per vnet.
"kubernetesConfig": {
"maxPods": 50
}
Using the default configuration, Kubernetes allows communication between all Pods within a cluster. To ensure that Pods can only be accessed by authorized Pods, a policy enforcement is needed. To enable policy enforcement using Calico refer to the cluster definition document under networkPolicy. There is also a reference cluster definition available here.
This will deploy a Calico node controller to every instance of the cluster using a Kubernetes DaemonSet. After a successful deployment you should be able to see these Pods running in your cluster:
kubectl get pods --namespace kube-system -l k8s-app=calico-node -o wide
NAME READY STATUS RESTARTS AGE IP NODE
calico-node-034zh 2/2 Running 0 2h 10.240.255.5 k8s-master-30179930-0
calico-node-qmr7n 2/2 Running 0 2h 10.240.0.4 k8s-agentpool1-30179930-1
calico-node-z3p02 2/2 Running 0 2h 10.240.0.5 k8s-agentpool1-30179930-0
Per default Calico still allows all communication within the cluster. Using Kubernetes' NetworkPolicy API, you can define stricter policies. Good resources to get information about that are:
ACS Engine supports deploying into an existing VNET. Operators must specify the ARM path/id of Subnets for the masterProfile and any agentPoolProfiles, as well as the first IP address to use for static IP allocation in firstConsecutiveStaticIP. Please note that in any azure subnet, the first four and the last ip address is reserved and can not be used. Additionally, each POD now gets the IP address from the Subnet. As a result, for the master nodes, enough IP addresses (equal to ipAddressCount) should be available beyond firstConsecutiveStaticIP. By default, the ipAddressCount has a value of 30, and can be changed if desired. Furthermore, to prevent source address NAT'ing within the VNET, we assign to the vnetCidr property in masterProfile the CIDR block that represents the usable address space in the existing VNET.
See below profiles as an example:
"masterProfile": {
...
"vnetSubnetId": "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/MASTER_SUBNET_NAME",
"firstConsecutiveStaticIP": "10.239.0.5",
"vnetCidr": "10.239.0.0/16",
...
},
...
"agentPoolProfiles": [
{
...
"name": "agentpri",
"vnetSubnetId": "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/AGENT_SUBNET_NAME",
...
},If you're not using Azure CNI (e.g., "networkPolicy": "none" in the kubernetesConfig api model configuration object): After a custom VNET-configured cluster finishes provisioning, fetch the id of the Route Table resource from Microsoft.Network provider in your new cluster's Resource Group.
The route table resource id is of the format: /subscriptions/SUBSCRIPTIONID/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.Network/routeTables/ROUTETABLENAME
Existing subnets will need to use the Kubernetes-based Route Table so that machines can route to Kubernetes-based workloads.
Update properties of all subnets in the existing VNET he route table resource by appending the following to subnet properties:
"routeTable": {
"id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Network/routeTables/k8s-master-<SOMEID>-routetable>"
}E.g.:
"subnets": [
{
"name": "subnetname",
"id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Network/virtualNetworks/<VirtualNetworkName>/subnets/<SubnetName>",
"properties": {
"provisioningState": "Succeeded",
"addressPrefix": "10.240.0.0/16",
"routeTable": {
"id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Network/routeTables/k8s-master-<SOMEID>-routetable"
}
...
}
...
}
]