HPE CSI Driver - April 2020.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HPE CSI DRIVER FOR KUBERNETES \n",
    "created by Dirk Derichsweiler / k8s@hpe.com\n",
    "& Stephan Koch / stephan.koch@hpe.com\n",
    "/ April 2020\n",
    "\n",
    "you can find this jupyter notebook here:\n",
    "https://github.com/dderichswei/csi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Container Storage Interface (CSI) Driver for Kubernetes. The HPE CSI Driver for Kubernetes allows you to use a Container Storage Provider (CSP) to perform data management operations on storage resources. The architecture of the CSI driver allows block storage vendors to implement a CSP that follows the spec (a browser friendly version).\n",
    "\n",
    "It allows a complete separation of concerns between upstream Kubernetes core, SIG Storage (CSI owners), CSI driver author (HPE) and the backend CSP developer.\n",
    "\n",
    "\n",
    "for more Information: https://scod.hpedev.io/csi_driver/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"pictures/1.png\" alt=\"SC\" width=\"700\" height=\"700\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## select the right cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:15:55.172158Z",
     "start_time": "2020-04-30T12:15:55.056767Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "azure-ddk8s                      hpecp-DD_PLAYGROUND-admin-config\n",
      "bigmama-k3s                      hpecp-ddtestcluster-admin-config\n",
      "ddk1-ddk8s-config                hpecp_hpc_demo\n",
      "DESKTOP-V6NAVHT-minikube-config  nimbletest_kubeconfig\n",
      "hcp_barem_sy480                  rancher-demo02-centos\n",
      "hpecp_csi_demo_admin_kubeconfig  rancher-rke-centos\n",
      "hpecp-DDK8S-HPECP-admin-config\n"
     ]
    }
   ],
   "source": [
    "ls ~/k8sconfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:16:11.042066Z",
     "start_time": "2020-04-30T12:16:10.936058Z"
    }
   },
   "outputs": [],
   "source": [
    "export KUBECONFIG=~/k8sconfig/hpecp_hpc_demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:16:34.801230Z",
     "start_time": "2020-04-30T12:16:34.198480Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mKubernetes master\u001b[0m is running at \u001b[0;33mhttps://ddcp5.container.demo.local:10004\u001b[0m\n",
      "\u001b[0;32mKubeDNS\u001b[0m is running at \u001b[0;33mhttps://ddcp5.container.demo.local:10004/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\u001b[0m\n",
      "\u001b[0;32mMetrics-server\u001b[0m is running at \u001b[0;33mhttps://ddcp5.container.demo.local:10004/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy\u001b[0m\n",
      "\n",
      "To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n",
      "NAME                          STATUS   ROLES    AGE     VERSION\n",
      "ddcp14.container.demo.local   Ready    master   2d20h   v1.17.0\n",
      "ddcp15.container.demo.local   Ready    worker   2d20h   v1.17.0\n"
     ]
    }
   ],
   "source": [
    "kubectl cluster-info\n",
    "kubectl get nodes\n",
    "#kubectl get pods -A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install with Helm\n",
    "\n",
    "other installation methods: \n",
    "- Object Configuration Files (https://github.com/hpe-storage/csi-driver)\n",
    "- Helm Charts (https://hub.helm.sh)\n",
    "- Operator (https://operatorhub.io)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:18:14.986690Z",
     "start_time": "2020-04-30T12:18:14.814789Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version.BuildInfo{Version:\"v3.0.2\", GitCommit:\"19e47ee3283ae98139d98460de796c1be1e3975f\", GitTreeState:\"clean\", GoVersion:\"go1.13.5\"}\n"
     ]
    }
   ],
   "source": [
    "helm version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:18:39.006654Z",
     "start_time": "2020-04-30T12:18:38.554227Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"hpe-storage\" has been added to your repositories\n"
     ]
    }
   ],
   "source": [
    "helm repo add hpe-storage https://hpe-storage.github.io/co-deployments/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:18:49.483312Z",
     "start_time": "2020-04-30T12:18:43.306088Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hang tight while we grab the latest from your chart repositories...\n",
      "...Successfully got an update from the \"hpe-storage\" chart repository\n",
      "...Successfully got an update from the \"oteemo\" chart repository\n",
      "...Successfully got an update from the \"vmware-tanzu\" chart repository\n",
      "...Successfully got an update from the \"jetstack\" chart repository\n",
      "...Successfully got an update from the \"harbor\" chart repository\n",
      "...Successfully got an update from the \"halkeye\" chart repository\n",
      "...Successfully got an update from the \"bitnami\" chart repository\n",
      "...Successfully got an update from the \"stable\" chart repository\n",
      "Update Complete. ⎈ Happy Helming!⎈ \n",
      "NAME                      \tCHART VERSION\tAPP VERSION\tDESCRIPTION                                       \n",
      "hpe-storage/hpe-csi-driver\t1.2.0        \t1.1.1      \tA Helm chart for installing the HPE CSI Driver ...\n"
     ]
    }
   ],
   "source": [
    "helm repo update\n",
    "helm search repo hpe-csi-driver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helm Parameters\n",
    "\n",
    "|Parameter|\tDescription\t|Default|\n",
    "|:--------|:-----------:|:------|\n",
    "|backendType\t|Name of the HPE Storage backend type (nimble, hpe3parprimera)|\tnimble|\n",
    "|secret.create\t|Enabled creation of secret along with driver deployment\t|true|\n",
    "|secret.backend\t|HPE storage backend hostname or IP address.\t|192.168.1.1|\n",
    "|secret.username|\tUsername for the backend.|\tadmin|\n",
    "|secret.password|\tPassword for the backend.\t|admin|\n",
    "|crd.nodeInfo.create|\tCreate nodeinfo CRDs required by HPE CSI driver. Should only enable with HELM 2, as they are automatically created with HELM 3 without this flag.|\tfalse|\n",
    "|crd.volumeInfo.create|\tCreate volumeinfo CRDs required by HPE CSI driver for 3PAR Primera. Should only enable with HELM 2, as they are automatically created with HELM 3 without this flag.|\tfalse|\n",
    "|logLevel|\tLog level. Can be one of info, debug, trace, warn and error|\tinfo|\n",
    "|imagePullPolicy\t|Image pull policy (Always, IfNotPresent, Never).\t|Always|\n",
    "|storageClass.name\t|The name to assign the created StorageClass.|\thpe-standard|\n",
    "|storageClass.create|\tEnables creation of StorageClass to consume this hpe-csi-driver instance\t|true|\n",
    "|storageClass.defaultClass|\tWhether to set the created StorageClass as the clusters default StorageClass.\t|false|\n",
    "|storageClass.parameters.fsType|\tType of file system being used (ext4, ext3, xfs, btrfs)\t|xfs|\n",
    "|storageClass.parameters.volumeDescription|\tVolume description for volumes created using HPE CSI driver|\t-|\n",
    "|storageClass.parameters.accessProtocol|\tAccess protocol to use for storage connectivity (iscsi, fc)\t|iscsi|\n",
    "|storageClass.parameters.provisioningType|\tProvisioning type for HPE 3PAR Primera\t|tpvv|\n",
    "|storageClass.parameters.cpg|\tCPG type for HPE 3PAR Primera\t|FC_r6|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-27T16:31:21.167788Z",
     "start_time": "2020-04-27T16:31:18.057892Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "helm install hpe-csi hpe-storage/hpe-csi-driver --namespace kube-system --set secret.backend=<nimbleip> --set secret.username=<nimbleuser> --set secret.password=<nimblepassword>\n",
    "#helm uninstall hpe-csi --namespace kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For FibreChannel Access:**\n",
    "\n",
    "Above we installed for the iscsi protocol.\n",
    "For FC use:\n",
    "\n",
    "``helm install hpe-csi hpe-storage/hpe-csi-driver --namespace kube-system --set storageClass.parameters.accessProtocol=fc --set secret.backend=<nimbleip> --set secret.username=<user> --set secret.password=<PW>``\n",
    "\n",
    "and do manually the FC Zoning.\n",
    "\n",
    "Adjust the accessProtocol for the StorageClass from iscsi to fc\n",
    "\n",
    "``accessProtocol: fc``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:20:53.200297Z",
     "start_time": "2020-04-30T12:20:53.010526Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME   \tNAMESPACE  \tREVISION\tUPDATED                                 \tSTATUS  \tCHART               \tAPP VERSION\n",
      "hpe-csi\tkube-system\t1       \t2020-04-27 18:31:20.665560775 +0200 CEST\tdeployed\thpe-csi-driver-1.2.0\t1.1.1      \n"
     ]
    }
   ],
   "source": [
    "helm list --all-namespaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:21:09.660020Z",
     "start_time": "2020-04-30T12:21:09.459076Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                                  READY   STATUS    RESTARTS   AGE\n",
      "canal-khh4s                                           3/3     Running   26         2d20h\n",
      "canal-lsfjb                                           3/3     Running   4          2d20h\n",
      "coredns-6955765f44-bjjcn                              1/1     Running   1          2d20h\n",
      "coredns-6955765f44-g2lx6                              1/1     Running   1          2d20h\n",
      "etcd-ddcp14.container.demo.local                      1/1     Running   1          2d20h\n",
      "hpe-csi-controller-84d8569476-xslj7                   5/5     Running   10         2d19h\n",
      "hpe-csi-node-zkppt                                    2/2     Running   5          2d19h\n",
      "kube-apiserver-ddcp14.container.demo.local            1/1     Running   1          2d20h\n",
      "kube-controller-manager-ddcp14.container.demo.local   1/1     Running   3          2d20h\n",
      "kube-proxy-6fwxb                                      1/1     Running   1          2d20h\n",
      "kube-proxy-bfbmd                                      1/1     Running   1          2d20h\n",
      "kube-scheduler-ddcp14.container.demo.local            1/1     Running   3          2d20h\n",
      "kube-state-metrics-55489476b6-d9w99                   1/1     Running   1          2d20h\n",
      "metricbeat-9sgnt                                      1/1     Running   5          2d20h\n",
      "metricbeat-dd949d58d-pwfm7                            1/1     Running   1          2d20h\n",
      "metricbeat-fg946                                      1/1     Running   1          2d20h\n",
      "metrics-server-66d84f9765-qlkm5                       1/1     Running   1          2d20h\n",
      "nimble-csp-546c9c4dd4-wqclk                           1/1     Running   1          2d19h\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods -n kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:22:00.620624Z",
     "start_time": "2020-04-30T12:22:00.220775Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\n",
      "default (default)   com.mapr.csi-kdf   Delete          Immediate           false                  2d20h\n",
      "hpe-standard        csi.hpe.com        Delete          Immediate           true                   2d19h\n",
      "---\n",
      "allowVolumeExpansion: true\n",
      "apiVersion: storage.k8s.io/v1\n",
      "kind: StorageClass\n",
      "metadata:\n",
      "  creationTimestamp: \"2020-04-27T16:31:22Z\"\n",
      "  labels:\n",
      "    plugin: hpe-csi\n",
      "  name: hpe-standard\n",
      "  resourceVersion: \"5860\"\n",
      "  selfLink: /apis/storage.k8s.io/v1/storageclasses/hpe-standard\n",
      "  uid: 4a1f8b6b-adc4-4379-bac5-a14d466e6d83\n",
      "parameters:\n",
      "  accessProtocol: iscsi\n",
      "  csi.storage.k8s.io/controller-expand-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/controller-publish-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/fstype: xfs\n",
      "  csi.storage.k8s.io/node-publish-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/node-publish-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/node-stage-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/node-stage-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/provisioner-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/provisioner-secret-namespace: kube-system\n",
      "  description: Volume created by the HPE CSI Driver for Kubernetes\n",
      "provisioner: csi.hpe.com\n",
      "reclaimPolicy: Delete\n",
      "volumeBindingMode: Immediate\n"
     ]
    }
   ],
   "source": [
    "kubectl get sc\n",
    "echo \"---\"\n",
    "kubectl get sc hpe-standard -o yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# create Secret\n",
    "\n",
    "nimble-secret is created with them \"Helm Installation\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:23:33.718615Z",
     "start_time": "2020-04-30T12:23:33.532823Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: v1\n",
      "data:\n",
      "  backend: MTAuMC4zMi4yNTM=\n",
      "  password: Q29tcGFxMSE=\n",
      "  serviceName: bmltYmxlLWNzcC1zdmM=\n",
      "  servicePort: ODA4MA==\n",
      "  username: YWRtaW4=\n",
      "kind: Secret\n",
      "metadata:\n",
      "  creationTimestamp: \"2020-04-27T16:31:22Z\"\n",
      "  name: nimble-secret\n",
      "  namespace: kube-system\n",
      "  resourceVersion: \"5858\"\n",
      "  selfLink: /api/v1/namespaces/kube-system/secrets/nimble-secret\n",
      "  uid: 2fb4f43b-9550-4044-a3c5-bb13ed5f5c10\n",
      "type: Opaque\n"
     ]
    }
   ],
   "source": [
    "kubectl get secret nimble-secret -n kube-system -o yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## create own Storageclass\n",
    "\n",
    "A StorageClass is used to provision or clone an HPE Nimble Storage-backed persistent volume. It can also be used to import an existing HPE Nimble Storage volume or clone of a snapshot into the Kubernetes cluster. The parameters are grouped below by those same workflows.\n",
    "\n",
    "``These are optional parameters unless specified.``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:24:17.651149Z",
     "start_time": "2020-04-30T12:24:17.419961Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\n",
      "default (default)   com.mapr.csi-kdf   Delete          Immediate           false                  2d20h\n",
      "hpe-standard        csi.hpe.com        Delete          Immediate           true                   2d19h\n"
     ]
    }
   ],
   "source": [
    "kubectl get sc\n",
    "#kubectl get pods -A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StorageClass parameters\n",
    "#### Common parameters for provisioning and cloning\n",
    "\n",
    "|Parameter|\tString\t|Description|\n",
    "|:--------|:-------:|:----------|\n",
    "|accessProtocol|\tText\t|The access protocol to use when accessing the persistent volume (\"fc\" or \"iscsi\"). Defaults to \"iscsi\" when unspecified.|\n",
    "|destroyOnDelete|\tBoolean\t|Indicates the backing Nimble volume (including snapshots) should be destroyed when the PVC is deleted.|\n",
    "|limitIops|\tInteger\t|The IOPS limit of the volume. The IOPS limit should be in the range 256 to 4294967294, or -1 for unlimited (default).|\n",
    "|limitMbps|\tInteger|\tThe MB/s throughput limit for the volume.|\n",
    "|description|\tText|\tText to be added to the volume's description on the Nimble array.|\n",
    "|performancePolicy|\tText|\tThe name of the performance policy to assign to the volume. Default example performance policies include \"Backup Repository\", \"Exchange 2003 data store\", \"Exchange 2007 data store\", \"Exchange 2010 data store\", \"Exchange log\", \"Oracle OLTP\", \"Other Workloads\", \"SharePoint\", \"SQL Server\", \"SQL Server 2012\", \"SQL Server Logs\".|\n",
    "|protectionTemplate|\tText\t|The name of the protection template to assign to the volume. Default examples of protection templates include \"Retain-30Daily\", \"Retain-48Hourly-30aily-52Weekly\", and \"Retain-90Daily\".|\n",
    "|folder\t|Text\t|The name of the Nimble folder in which to place the volume.|\n",
    "|thick\t|Boolean\t|Indicates that the volume should be thick provisioned.|\n",
    "|dedupeEnabled|\tBoolean\t|Indicates that the volume should enable deduplication.|\n",
    "|syncOnDetach|\tBoolean\t|Indicates that a snapshot of the volume should be synced to the replication partner each time it is detached from a node.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Provisioning parameters\n",
    "These parameters are immutable for clones once a volume has been created.\n",
    "\n",
    "``fsOwner, fsMode, and fsCreateOptions are not applicable when using volumeMode: Block in the PersistentVolumeClaim.``\n",
    "\n",
    "|Parameter\t|String\t|Description|\n",
    "|:----------|:-----:|:----------|\n",
    "|fsOwner\t|userId:groupId\t|The user id and group id that should own the root directory of the filesystem.|\n",
    "|fsMode\t|Octal digits|\t1 to 4 octal digits that represent the file mode to be applied to the root directory of the filesystem.|\n",
    "|fsCreateOptions|\tText\t|A string to be passed to the mkfs command. These flags are opaque to CSI and are therefore not validated. To protect the node, only the following characters are allowed: [a-zA-Z0-9=, \\-].|\n",
    "|encrypted|\tBoolean\t|Indicates that the volume should be encrypted.|\n",
    "|pool\t|Text|\tThe name of the pool in which to place the volume.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pod inline volume parameters (Local Ephemeral Volumes)\n",
    "These parameters are applicable only for Pod inline volumes and to be specified within Pod spec.\n",
    "\n",
    "``All parameters are required for inline ephemeral volumes.``\n",
    "\n",
    "|Parameter\t|String\t|Description|\n",
    "|:----------|:-----:|:----------|\n",
    "|csi.storage.k8s.io/ephemeral\t|Boolean\t|Indicates that the request is for ephemeral inline volume. This is a mandatory parameter and must be set to \"true\".|\n",
    "|inline-volume-secret-name|\tText\t|A reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume call.|\n",
    "|inline-volume-secret-namespace\t|Text\t|The namespace of inline-volume-secret-name for ephemeral inline volume.|\n",
    "|size\t|Text\t|The size of ephemeral volume specified in MiB or GiB. If unspecified, a default value will be used.|\n",
    "|accessProtocol|\tText\t|Storage access protocol to use, \"iscsi\" or \"fc\".|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cloning parameters\n",
    "\n",
    "Cloning supports two modes of cloning. Either use cloneOf and reference a PVC in the current namespace or use importVolAsClone and reference a Nimble volume name to clone and import to Kubernetes.\n",
    "\n",
    "|Parameter\t|String\t|Description|\n",
    "|:----------|:-----:|:----------|\n",
    "|cloneOf\t|Text\t|The name of the PV to be cloned. cloneOf and importVolAsClone are mutually exclusive.|\n",
    "|importVolAsClone|\tText\t|The name of the Nimble volume to clone and import. importVolAsClone and cloneOf are mutually exclusive.|\n",
    "|snapshot|\tText\t|The name of the snapshot to base the clone on. This is optional. If not specified, a new snapshot is created.|\n",
    "|createSnapshot|\tBoolean\t|Indicates that a new snapshot of the volume should be taken matching the name provided in the snapshot parameter. If the snapshot parameter is not specified, a default name will be created.|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import parameters\n",
    "\n",
    "Importing volumes to Kubernetes requires the source Nimble volume to be offline. In case of reverse replication, the upstream volume should be in offline state. All previous Access Control Records and Initiator Groups will be stripped from the volume when put under control of the HPE CSI Driver.\n",
    "\n",
    "|Parameter\t|String\t|Description|\n",
    "|:----------|:-----:|:----------|\n",
    "|importVolumeName\t|Text\t|The name of the Nimble volume to import.|\n",
    "|snapshot|\tText|\tThe name of the Nimble snapshot to restore the imported volume to after takeover. If not specified, the volume will not be restored.|\n",
    "|takeover|\tBoolean\t|Indicates the current group will takeover ownership of the Nimble volume and volume collection. This should be performed against a downstream replica.|\n",
    "|reverseReplication\t|Boolean\t|Reverses the replication direction so that writes to the Nimble volume are replicated back to the group where it was replicated from.|\n",
    "|forceImport\t|Boolean\t|Forces the import of a volume that is not owned by the group and is not part of a volume collection. If the volume is part of a volume collection, use takeover instead.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### VolumeSnapshotClass parameters\n",
    "\n",
    "These parametes are for VolumeSnapshotClass objects when using CSI snapshots. Please see using CSI snapshots for more details.\n",
    "\n",
    "|Parameter\t|String\t|Description|\n",
    "|:----------|:-----:|:----------|\n",
    "|description\t|Text\t|Text to be added to the snapshot's description on the Nimble array.|\n",
    "|writable\t|Boolean\t|Indicates if the snapshot is writable on the Nimble array.|\n",
    "|online\t|Boolean\t|Indicates if the snapshot is set to online on the Nimble array.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make container persistent with the HPE CSI Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T13:03:18.720344Z",
     "start_time": "2020-04-30T13:03:18.613034Z"
    }
   },
   "source": [
    "<img src=\"pictures/2.png\" alt=\"SC\" width=\"700\" height=\"700\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## check and set the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:25:56.111229Z",
     "start_time": "2020-04-30T12:25:55.580503Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                          STATUS   ROLES    AGE     VERSION\n",
      "ddcp14.container.demo.local   Ready    master   2d20h   v1.17.0\n",
      "ddcp15.container.demo.local   Ready    worker   2d20h   v1.17.0\n",
      "--- \n",
      "NAME                PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\n",
      "default (default)   com.mapr.csi-kdf   Delete          Immediate           false                  2d20h\n",
      "hpe-standard        csi.hpe.com        Delete          Immediate           true                   2d19h\n",
      "--- \n",
      "CURRENT   NAME                      CLUSTER   AUTHINFO           NAMESPACE\n",
      "*         kubernetes-admin@k8s-16   k8s-16    kubernetes-admin   simple-demo\n"
     ]
    }
   ],
   "source": [
    "kubectl get nodes\n",
    "echo \"--- \"\n",
    "kubectl get sc\n",
    "echo \"--- \"\n",
    "kubectl config get-contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:26:36.940415Z",
     "start_time": "2020-04-30T12:26:36.283249Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/simple-demo created\n",
      "Context \"kubernetes-admin@k8s-16\" modified.\n",
      "No resources found in simple-demo namespace.\n"
     ]
    }
   ],
   "source": [
    "kubectl create ns simple-demo\n",
    "kubectl config set-context --current --namespace=simple-demo\n",
    "kubectl get all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-22T14:20:35.817904Z",
     "start_time": "2020-04-22T14:20:35.711819Z"
    }
   },
   "source": [
    "## create a Storageclass\n",
    "\n",
    "\n",
    "<img src=\"pictures/sc.png\" alt=\"SC\" width=\"350\" height=\"350\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:30:10.990565Z",
     "start_time": "2020-04-30T12:30:09.091693Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "storageclass.storage.k8s.io/ddnimble created\n"
     ]
    }
   ],
   "source": [
    "#Storageclass\n",
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "kind: StorageClass\n",
    "apiVersion: storage.k8s.io/v1\n",
    "metadata:\n",
    "  name: ddnimble\n",
    "provisioner: csi.hpe.com\n",
    "allowVolumeExpansion: true\n",
    "parameters:\n",
    "  accessProtocol: iscsi\n",
    "  csi.storage.k8s.io/controller-expand-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/controller-publish-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/fstype: xfs\n",
    "  csi.storage.k8s.io/node-publish-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/node-publish-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/node-stage-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/node-stage-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/provisioner-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/provisioner-secret-namespace: kube-system\n",
    "  description: Volume created by the HPE CSI Driver for Kubernetes\n",
    "  folder: \"simple-demo\"\n",
    "  reclaimPolicy: Delete\n",
    "  volumeBindingMode: Immediate\n",
    "  destroyOnDelete: \"true\"\n",
    "  limitIops: \"38400\"\n",
    "  limitMbps: \"2048\"\n",
    "  allowOverrides: description, limitIops, limitMbps, folder, destroyOnDelete\n",
    "\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:30:16.509533Z",
     "start_time": "2020-04-30T12:30:16.126494Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\n",
      "ddnimble            csi.hpe.com        Delete          Immediate           true                   6s\n",
      "default (default)   com.mapr.csi-kdf   Delete          Immediate           false                  2d20h\n",
      "hpe-standard        csi.hpe.com        Delete          Immediate           true                   2d19h\n",
      "---\n",
      "allowVolumeExpansion: true\n",
      "apiVersion: storage.k8s.io/v1\n",
      "kind: StorageClass\n",
      "metadata:\n",
      "  annotations:\n",
      "    kubectl.kubernetes.io/last-applied-configuration: |\n",
      "      {\"allowVolumeExpansion\":true,\"apiVersion\":\"storage.k8s.io/v1\",\"kind\":\"StorageClass\",\"metadata\":{\"annotations\":{},\"name\":\"ddnimble\"},\"parameters\":{\"accessProtocol\":\"iscsi\",\"allowOverrides\":\"description, limitIops, limitMbps, folder, destroyOnDelete\",\"csi.storage.k8s.io/controller-expand-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/controller-expand-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/controller-publish-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/controller-publish-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/fstype\":\"xfs\",\"csi.storage.k8s.io/node-publish-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/node-publish-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/node-stage-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/node-stage-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/provisioner-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/provisioner-secret-namespace\":\"kube-system\",\"description\":\"Volume created by the HPE CSI Driver for Kubernetes\",\"destroyOnDelete\":\"true\",\"folder\":\"simple-demo\",\"limitIops\":\"38400\",\"limitMbps\":\"2048\",\"reclaimPolicy\":\"Delete\",\"volumeBindingMode\":\"Immediate\"},\"provisioner\":\"csi.hpe.com\"}\n",
      "  creationTimestamp: \"2020-04-30T12:30:10Z\"\n",
      "  name: ddnimble\n",
      "  resourceVersion: \"540252\"\n",
      "  selfLink: /apis/storage.k8s.io/v1/storageclasses/ddnimble\n",
      "  uid: d791d69e-7963-4f0b-811b-4186d1f290ca\n",
      "parameters:\n",
      "  accessProtocol: iscsi\n",
      "  allowOverrides: description, limitIops, limitMbps, folder, destroyOnDelete\n",
      "  csi.storage.k8s.io/controller-expand-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/controller-publish-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/fstype: xfs\n",
      "  csi.storage.k8s.io/node-publish-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/node-publish-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/node-stage-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/node-stage-secret-namespace: kube-system\n",
      "  csi.storage.k8s.io/provisioner-secret-name: nimble-secret\n",
      "  csi.storage.k8s.io/provisioner-secret-namespace: kube-system\n",
      "  description: Volume created by the HPE CSI Driver for Kubernetes\n",
      "  destroyOnDelete: \"true\"\n",
      "  folder: simple-demo\n",
      "  limitIops: \"38400\"\n",
      "  limitMbps: \"2048\"\n",
      "  reclaimPolicy: Delete\n",
      "  volumeBindingMode: Immediate\n",
      "provisioner: csi.hpe.com\n",
      "reclaimPolicy: Delete\n",
      "volumeBindingMode: Immediate\n"
     ]
    }
   ],
   "source": [
    "kubectl get sc\n",
    "echo \"---\"\n",
    "kubectl get sc ddnimble -o yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## create Persistent Volume Claim (User request)\n",
    "\n",
    "\n",
    "<img src=\"pictures/pvc.png\" alt=\"SC\" width=\"350\" height=\"350\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:31:55.951285Z",
     "start_time": "2020-04-30T12:31:54.791809Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "persistentvolumeclaim/ddpvc1 created\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "kind: PersistentVolumeClaim\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: ddpvc1\n",
    "  annotations:\n",
    "    csi.hpe.com/description: \"demo the nimble plugin for K8S/Openshift/Docker/Rancher\"\n",
    "    csi.hpe.com/limitIOPS: \"8000\"\n",
    "    csi.hpe.com/destroyOnDelete: \"true\" \n",
    "spec:\n",
    "  accessModes:\n",
    "    - ReadWriteOnce\n",
    "  resources:\n",
    "    requests:\n",
    "      storage: 5Gi\n",
    "  storageClassName: ddnimble\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:32:26.554711Z",
     "start_time": "2020-04-30T12:32:25.788282Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\n",
      "ddpvc1   Bound    pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            ddnimble       30s\n",
      "---\n",
      "NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE\n",
      "pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            Delete           Bound    simple-demo/ddpvc1   ddnimble                30s\n",
      "---\n",
      "Name:          ddpvc1\n",
      "Namespace:     simple-demo\n",
      "StorageClass:  ddnimble\n",
      "Status:        Bound\n",
      "Volume:        pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1\n",
      "Labels:        <none>\n",
      "Annotations:   csi.hpe.com/description: demo the nimble plugin for K8S/Openshift/Docker/Rancher\n",
      "               csi.hpe.com/destroyOnDelete: true\n",
      "               csi.hpe.com/limitIOPS: 8000\n",
      "               kubectl.kubernetes.io/last-applied-configuration:\n",
      "                 {\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"csi.hpe.com/description\":\"demo the nimble plugin for K8S/Ope...\n",
      "               pv.kubernetes.io/bind-completed: yes\n",
      "               pv.kubernetes.io/bound-by-controller: yes\n",
      "               volume.beta.kubernetes.io/storage-provisioner: csi.hpe.com\n",
      "Finalizers:    [kubernetes.io/pvc-protection]\n",
      "Capacity:      5Gi\n",
      "Access Modes:  RWO\n",
      "VolumeMode:    Filesystem\n",
      "Mounted By:    <none>\n",
      "Events:\n",
      "  Type    Reason                 Age                From                                                                          Message\n",
      "  ----    ------                 ----               ----                                                                          -------\n",
      "  Normal  ExternalProvisioning   31s (x2 over 31s)  persistentvolume-controller                                                   waiting for a volume to be created, either by external provisioner \"csi.hpe.com\" or manually created by system administrator\n",
      "  Normal  Provisioning           31s                csi.hpe.com_ddcp15.container.demo.local_eefdabc5-56fa-4b73-a527-2026440375a1  External provisioner is provisioning volume for claim \"simple-demo/ddpvc1\"\n",
      "  Normal  ProvisioningSucceeded  30s                csi.hpe.com_ddcp15.container.demo.local_eefdabc5-56fa-4b73-a527-2026440375a1  Successfully provisioned volume pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1\n",
      "---\n",
      "Name:            pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1\n",
      "Labels:          <none>\n",
      "Annotations:     pv.kubernetes.io/provisioned-by: csi.hpe.com\n",
      "Finalizers:      [kubernetes.io/pv-protection]\n",
      "StorageClass:    ddnimble\n",
      "Status:          Bound\n",
      "Claim:           simple-demo/ddpvc1\n",
      "Reclaim Policy:  Delete\n",
      "Access Modes:    RWO\n",
      "VolumeMode:      Filesystem\n",
      "Capacity:        5Gi\n",
      "Node Affinity:   <none>\n",
      "Message:         \n",
      "Source:\n",
      "    Type:              CSI (a Container Storage Interface (CSI) volume source)\n",
      "    Driver:            csi.hpe.com\n",
      "    VolumeHandle:      063fd5154c4dd89fc9000000000000000000000148\n",
      "    ReadOnly:          false\n",
      "    VolumeAttributes:      accessProtocol=iscsi\n",
      "                           allowOverrides=description, limitIops, limitMbps, folder, destroyOnDelete\n",
      "                           description=demo the nimble plugin for K8S/Openshift/Docker/Rancher\n",
      "                           destroyOnDelete=true\n",
      "                           folder=simple-demo\n",
      "                           fsType=xfs\n",
      "                           limitIops=8000\n",
      "                           limitMbps=2048\n",
      "                           reclaimPolicy=Delete\n",
      "                           storage.kubernetes.io/csiProvisionerIdentity=1588191545517-8081-csi.hpe.com\n",
      "                           volumeAccessMode=mount\n",
      "                           volumeBindingMode=Immediate\n",
      "Events:                <none>\n"
     ]
    }
   ],
   "source": [
    "kubectl get pvc\n",
    "echo \"---\"\n",
    "kubectl get pv\n",
    "echo \"---\"\n",
    "kubectl describe pvc ddpvc1\n",
    "echo \"---\"\n",
    "kubectl describe pv "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## create Pod and attach Persistent Volume Claim\n",
    "\n",
    "<img src=\"pictures/podpvc.png\" alt=\"SC\" width=\"350\" height=\"350\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:34:37.282015Z",
     "start_time": "2020-04-30T12:34:36.067920Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/ddpod1 created\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "kind: Pod\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: ddpod1\n",
    "spec:\n",
    "  containers:\n",
    "  - name: nginx\n",
    "    image: nginx\n",
    "    volumeMounts:\n",
    "    - name: export\n",
    "      mountPath: /export\n",
    "  restartPolicy: Always\n",
    "  volumes:\n",
    "  - name: export\n",
    "    persistentVolumeClaim:\n",
    "      claimName: ddpvc1\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:35:19.256290Z",
     "start_time": "2020-04-30T12:35:18.925010Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS    RESTARTS   AGE\n",
      "ddpod1   1/1     Running   0          42s\n",
      "Name:         ddpod1\n",
      "Namespace:    simple-demo\n",
      "Priority:     0\n",
      "Node:         ddcp15.container.demo.local/10.0.31.97\n",
      "Start Time:   Thu, 30 Apr 2020 14:34:37 +0200\n",
      "Labels:       <none>\n",
      "Annotations:  cni.projectcalico.org/podIP: 10.192.0.187/32\n",
      "              kubectl.kubernetes.io/last-applied-configuration:\n",
      "                {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"ddpod1\",\"namespace\":\"simple-demo\"},\"spec\":{\"containers\":[{\"image\":\"ng...\n",
      "              kubernetes.io/psp: 00-privileged\n",
      "Status:       Running\n",
      "IP:           10.192.0.187\n",
      "IPs:\n",
      "  IP:  10.192.0.187\n",
      "Containers:\n",
      "  nginx:\n",
      "    Container ID:   docker://c6f54b9798dd9f3f92e02264521c0217732f43467c25af3b948823ea16479f05\n",
      "    Image:          nginx\n",
      "    Image ID:       docker-pullable://docker.io/nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12\n",
      "    Port:           <none>\n",
      "    Host Port:      <none>\n",
      "    State:          Running\n",
      "      Started:      Thu, 30 Apr 2020 14:34:57 +0200\n",
      "    Ready:          True\n",
      "    Restart Count:  0\n",
      "    Environment:    <none>\n",
      "    Mounts:\n",
      "      /export from export (rw)\n",
      "      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mq9qx (ro)\n",
      "Conditions:\n",
      "  Type              Status\n",
      "  Initialized       True \n",
      "  Ready             True \n",
      "  ContainersReady   True \n",
      "  PodScheduled      True \n",
      "Volumes:\n",
      "  export:\n",
      "    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n",
      "    ClaimName:  ddpvc1\n",
      "    ReadOnly:   false\n",
      "  default-token-mq9qx:\n",
      "    Type:        Secret (a volume populated by a Secret)\n",
      "    SecretName:  default-token-mq9qx\n",
      "    Optional:    false\n",
      "QoS Class:       BestEffort\n",
      "Node-Selectors:  <none>\n",
      "Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n",
      "                 node.kubernetes.io/unreachable:NoExecute for 300s\n",
      "Events:\n",
      "  Type    Reason                  Age   From                                  Message\n",
      "  ----    ------                  ----  ----                                  -------\n",
      "  Normal  Scheduled               42s   default-scheduler                     Successfully assigned simple-demo/ddpod1 to ddcp15.container.demo.local\n",
      "  Normal  SuccessfulAttachVolume  42s   attachdetach-controller               AttachVolume.Attach succeeded for volume \"pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1\"\n",
      "  Normal  Pulling                 24s   kubelet, ddcp15.container.demo.local  Pulling image \"nginx\"\n",
      "  Normal  Pulled                  22s   kubelet, ddcp15.container.demo.local  Successfully pulled image \"nginx\"\n",
      "  Normal  Created                 22s   kubelet, ddcp15.container.demo.local  Created container nginx\n",
      "  Normal  Started                 22s   kubelet, ddcp15.container.demo.local  Started container nginx\n"
     ]
    }
   ],
   "source": [
    "kubectl get pod\n",
    "kubectl describe pod ddpod1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## let´s have a look inside of the container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:36:17.630566Z",
     "start_time": "2020-04-30T12:36:17.361032Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ddpod1.txt\n"
     ]
    }
   ],
   "source": [
    "kubectl exec pod/ddpod1 ls /export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## change some files inside the container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:36:14.978786Z",
     "start_time": "2020-04-30T12:36:14.477205Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Apr 30 12:36:14 UTC 2020\n"
     ]
    }
   ],
   "source": [
    "kubectl exec ddpod1 -- bash -c \"date > /export/ddpod1.txt\" \n",
    "kubectl exec ddpod1 cat /export/ddpod1.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## destroy the pod1, create an second, with volume of first pod\n",
    "\n",
    "<img src=\"pictures/delpod.png\" alt=\"SC\" width=\"550\" height=\"550\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:36:48.186232Z",
     "start_time": "2020-04-30T12:36:35.140163Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod \"ddpod1\" deleted\n"
     ]
    }
   ],
   "source": [
    "kubectl delete pod ddpod1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:36:48.547963Z",
     "start_time": "2020-04-30T12:36:48.190667Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No resources found in simple-demo namespace.\n"
     ]
    }
   ],
   "source": [
    "kubectl get pod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:36:55.028662Z",
     "start_time": "2020-04-30T12:36:54.843080Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE\n",
      "pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            Delete           Bound    simple-demo/ddpvc1   ddnimble                4m58s\n"
     ]
    }
   ],
   "source": [
    "kubectl get pv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## create a new Pod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:37:17.909410Z",
     "start_time": "2020-04-30T12:37:16.664541Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/ddpod2 created\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "kind: Pod\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: ddpod2\n",
    "spec:\n",
    "  containers:\n",
    "  - name: nginx\n",
    "    image: nginx\n",
    "    volumeMounts:\n",
    "    - name: export\n",
    "      mountPath: /export\n",
    "  restartPolicy: Always\n",
    "  volumes:\n",
    "  - name: export\n",
    "    persistentVolumeClaim:\n",
    "      claimName: ddpvc1\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:37:32.269637Z",
     "start_time": "2020-04-30T12:37:31.852621Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME         READY   STATUS    RESTARTS   AGE\n",
      "pod/ddpod2   1/1     Running   0          14s\n",
      "---\n",
      "Name:         ddpod2\n",
      "Namespace:    simple-demo\n",
      "Priority:     0\n",
      "Node:         ddcp15.container.demo.local/10.0.31.97\n",
      "Start Time:   Thu, 30 Apr 2020 14:37:17 +0200\n",
      "Labels:       <none>\n",
      "Annotations:  cni.projectcalico.org/podIP: 10.192.0.188/32\n",
      "              kubectl.kubernetes.io/last-applied-configuration:\n",
      "                {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"ddpod2\",\"namespace\":\"simple-demo\"},\"spec\":{\"containers\":[{\"image\":\"ng...\n",
      "              kubernetes.io/psp: 00-privileged\n",
      "Status:       Running\n",
      "IP:           10.192.0.188\n",
      "IPs:\n",
      "  IP:  10.192.0.188\n",
      "Containers:\n",
      "  nginx:\n",
      "    Container ID:   docker://7de2676e624d941792af512a81e4cc6901515569ff4b4b6d5bb8fa3f11923978\n",
      "    Image:          nginx\n",
      "    Image ID:       docker-pullable://docker.io/nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12\n",
      "    Port:           <none>\n",
      "    Host Port:      <none>\n",
      "    State:          Running\n",
      "      Started:      Thu, 30 Apr 2020 14:37:30 +0200\n",
      "    Ready:          True\n",
      "    Restart Count:  0\n",
      "    Environment:    <none>\n",
      "    Mounts:\n",
      "      /export from export (rw)\n",
      "      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mq9qx (ro)\n",
      "Conditions:\n",
      "  Type              Status\n",
      "  Initialized       True \n",
      "  Ready             True \n",
      "  ContainersReady   True \n",
      "  PodScheduled      True \n",
      "Volumes:\n",
      "  export:\n",
      "    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n",
      "    ClaimName:  ddpvc1\n",
      "    ReadOnly:   false\n",
      "  default-token-mq9qx:\n",
      "    Type:        Secret (a volume populated by a Secret)\n",
      "    SecretName:  default-token-mq9qx\n",
      "    Optional:    false\n",
      "QoS Class:       BestEffort\n",
      "Node-Selectors:  <none>\n",
      "Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n",
      "                 node.kubernetes.io/unreachable:NoExecute for 300s\n",
      "Events:\n",
      "  Type    Reason                  Age   From                                  Message\n",
      "  ----    ------                  ----  ----                                  -------\n",
      "  Normal  Scheduled               15s   default-scheduler                     Successfully assigned simple-demo/ddpod2 to ddcp15.container.demo.local\n",
      "  Normal  SuccessfulAttachVolume  14s   attachdetach-controller               AttachVolume.Attach succeeded for volume \"pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1\"\n",
      "  Normal  Pulling                 4s    kubelet, ddcp15.container.demo.local  Pulling image \"nginx\"\n",
      "  Normal  Pulled                  2s    kubelet, ddcp15.container.demo.local  Successfully pulled image \"nginx\"\n",
      "  Normal  Created                 2s    kubelet, ddcp15.container.demo.local  Created container nginx\n",
      "  Normal  Started                 2s    kubelet, ddcp15.container.demo.local  Started container nginx\n"
     ]
    }
   ],
   "source": [
    "kubectl get all\n",
    "echo \"---\"\n",
    "kubectl describe pod ddpod2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:17:16.457505Z",
     "start_time": "2020-04-30T09:17:16.272015Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:37:46.138830Z",
     "start_time": "2020-04-30T12:37:45.176641Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ddpod1.txt\n",
      "Thu Apr 30 12:36:14 UTC 2020\n",
      "Thu Apr 30 12:37:45 UTC 2020\n"
     ]
    }
   ],
   "source": [
    "kubectl exec ddpod2 ls /export\n",
    "kubectl exec ddpod2 cat /export/ddpod1.txt\n",
    "kubectl exec ddpod2 -- bash -c \"date > /export/ddpod2.txt\" \n",
    "kubectl exec ddpod2 cat /export/ddpod2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## clone the volume\n",
    "\n",
    "<img src=\"pictures/clone.png\" alt=\"SC\" width=\"550\" height=\"550\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:38:43.818595Z",
     "start_time": "2020-04-30T12:38:42.724945Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "persistentvolumeclaim/ddpvc1-clone created\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "  kind: PersistentVolumeClaim\n",
    "  apiVersion: v1\n",
    "  metadata:\n",
    "    name: ddpvc1-clone\n",
    "  spec:\n",
    "    storageClassName: ddnimble\n",
    "    dataSource:\n",
    "      name: ddpvc1\n",
    "      kind: PersistentVolumeClaim\n",
    "    accessModes:\n",
    "      - ReadWriteOnce\n",
    "    resources:\n",
    "      requests:\n",
    "        storage: 5Gi\n",
    "EOF\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:38:50.111167Z",
     "start_time": "2020-04-30T12:38:49.500631Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\n",
      "ddpvc1         Bound    pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            ddnimble       6m54s\n",
      "ddpvc1-clone   Bound    pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10   5Gi        RWO            ddnimble       6s\n",
      "---\n",
      "NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE\n",
      "pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10   5Gi        RWO            Delete           Bound    simple-demo/ddpvc1-clone   ddnimble                5s\n",
      "pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            Delete           Bound    simple-demo/ddpvc1         ddnimble                6m53s\n",
      "---\n",
      "Name:          ddpvc1-clone\n",
      "Namespace:     simple-demo\n",
      "StorageClass:  ddnimble\n",
      "Status:        Bound\n",
      "Volume:        pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10\n",
      "Labels:        <none>\n",
      "Annotations:   kubectl.kubernetes.io/last-applied-configuration:\n",
      "                 {\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"ddpvc1-clone\",\"namespace\":\"simple-demo\"},\"spec\":{\"a...\n",
      "               pv.kubernetes.io/bind-completed: yes\n",
      "               pv.kubernetes.io/bound-by-controller: yes\n",
      "               volume.beta.kubernetes.io/storage-provisioner: csi.hpe.com\n",
      "Finalizers:    [kubernetes.io/pvc-protection]\n",
      "Capacity:      5Gi\n",
      "Access Modes:  RWO\n",
      "VolumeMode:    Filesystem\n",
      "DataSource:\n",
      "  Kind:      PersistentVolumeClaim\n",
      "  Name:      ddpvc1\n",
      "Mounted By:  <none>\n",
      "Events:\n",
      "  Type    Reason                 Age   From                                                                          Message\n",
      "  ----    ------                 ----  ----                                                                          -------\n",
      "  Normal  ExternalProvisioning   7s    persistentvolume-controller                                                   waiting for a volume to be created, either by external provisioner \"csi.hpe.com\" or manually created by system administrator\n",
      "  Normal  Provisioning           7s    csi.hpe.com_ddcp15.container.demo.local_eefdabc5-56fa-4b73-a527-2026440375a1  External provisioner is provisioning volume for claim \"simple-demo/ddpvc1-clone\"\n",
      "  Normal  ProvisioningSucceeded  6s    csi.hpe.com_ddcp15.container.demo.local_eefdabc5-56fa-4b73-a527-2026440375a1  Successfully provisioned volume pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10\n"
     ]
    }
   ],
   "source": [
    "kubectl get pvc\n",
    "echo \"---\"\n",
    "kubectl get pv\n",
    "echo \"---\"\n",
    "kubectl describe pvc ddpvc1-clone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:39:21.025440Z",
     "start_time": "2020-04-30T12:39:19.795689Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/ddpod3 created\n"
     ]
    }
   ],
   "source": [
    "kubectl apply -f - << EOF\n",
    "---\n",
    "kind: Pod\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: ddpod3\n",
    "spec:\n",
    "  containers:\n",
    "  - name: nginx\n",
    "    image: nginx\n",
    "    volumeMounts:\n",
    "    - name: export\n",
    "      mountPath: /export\n",
    "  restartPolicy: Always\n",
    "  volumes:\n",
    "  - name: export\n",
    "    persistentVolumeClaim:\n",
    "      claimName: ddpvc1-clone\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:39:39.059489Z",
     "start_time": "2020-04-30T12:39:38.483122Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS    RESTARTS   AGE\n",
      "ddpod2   1/1     Running   0          2m21s\n",
      "ddpod3   1/1     Running   0          18s\n",
      "---\n",
      "NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\n",
      "ddpvc1         Bound    pvc-6eb3305f-3c71-4440-9fdf-841bbf52dcb1   5Gi        RWO            ddnimble       7m43s\n",
      "ddpvc1-clone   Bound    pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10   5Gi        RWO            ddnimble       55s\n",
      "---\n",
      "Name:         ddpod3\n",
      "Namespace:    simple-demo\n",
      "Priority:     0\n",
      "Node:         ddcp15.container.demo.local/10.0.31.97\n",
      "Start Time:   Thu, 30 Apr 2020 14:39:21 +0200\n",
      "Labels:       <none>\n",
      "Annotations:  cni.projectcalico.org/podIP: 10.192.0.189/32\n",
      "              kubectl.kubernetes.io/last-applied-configuration:\n",
      "                {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"ddpod3\",\"namespace\":\"simple-demo\"},\"spec\":{\"containers\":[{\"image\":\"ng...\n",
      "              kubernetes.io/psp: 00-privileged\n",
      "Status:       Running\n",
      "IP:           10.192.0.189\n",
      "IPs:\n",
      "  IP:  10.192.0.189\n",
      "Containers:\n",
      "  nginx:\n",
      "    Container ID:   docker://2509f99d8061d933db365cdda12760677ba4e9759e183f08a567a212ad4c050d\n",
      "    Image:          nginx\n",
      "    Image ID:       docker-pullable://docker.io/nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12\n",
      "    Port:           <none>\n",
      "    Host Port:      <none>\n",
      "    State:          Running\n",
      "      Started:      Thu, 30 Apr 2020 14:39:33 +0200\n",
      "    Ready:          True\n",
      "    Restart Count:  0\n",
      "    Environment:    <none>\n",
      "    Mounts:\n",
      "      /export from export (rw)\n",
      "      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mq9qx (ro)\n",
      "Conditions:\n",
      "  Type              Status\n",
      "  Initialized       True \n",
      "  Ready             True \n",
      "  ContainersReady   True \n",
      "  PodScheduled      True \n",
      "Volumes:\n",
      "  export:\n",
      "    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n",
      "    ClaimName:  ddpvc1-clone\n",
      "    ReadOnly:   false\n",
      "  default-token-mq9qx:\n",
      "    Type:        Secret (a volume populated by a Secret)\n",
      "    SecretName:  default-token-mq9qx\n",
      "    Optional:    false\n",
      "QoS Class:       BestEffort\n",
      "Node-Selectors:  <none>\n",
      "Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n",
      "                 node.kubernetes.io/unreachable:NoExecute for 300s\n",
      "Events:\n",
      "  Type    Reason                  Age   From                                  Message\n",
      "  ----    ------                  ----  ----                                  -------\n",
      "  Normal  Scheduled               18s   default-scheduler                     Successfully assigned simple-demo/ddpod3 to ddcp15.container.demo.local\n",
      "  Normal  SuccessfulAttachVolume  17s   attachdetach-controller               AttachVolume.Attach succeeded for volume \"pvc-32abcfef-9e4b-4b67-bc76-fcc17fc70d10\"\n",
      "  Normal  Pulling                 7s    kubelet, ddcp15.container.demo.local  Pulling image \"nginx\"\n",
      "  Normal  Pulled                  5s    kubelet, ddcp15.container.demo.local  Successfully pulled image \"nginx\"\n",
      "  Normal  Created                 5s    kubelet, ddcp15.container.demo.local  Created container nginx\n",
      "  Normal  Started                 5s    kubelet, ddcp15.container.demo.local  Started container nginx\n"
     ]
    }
   ],
   "source": [
    "kubectl get pod\n",
    "echo \"---\"\n",
    "kubectl get pvc\n",
    "echo \"---\"\n",
    "kubectl describe pod ddpod3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:39:51.492296Z",
     "start_time": "2020-04-30T12:39:50.988917Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ddpod1.txt\n",
      "ddpod2.txt\n",
      "Thu Apr 30 12:37:45 UTC 2020\n"
     ]
    }
   ],
   "source": [
    "kubectl exec ddpod3  ls /export\n",
    "kubectl exec ddpod3 cat /export/ddpod2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-27T13:06:24.687070Z",
     "start_time": "2020-04-27T13:06:24.580997Z"
    }
   },
   "source": [
    "## snapshot the volume\n",
    "\n",
    "https://scod.hpedev.io/csi_driver/using.html#enabling_csi_snapshots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the necessary resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As per the Kubernetes Special Interest Group (SIG) Storage, the snapshot controllers, custom resource definitions and RBAC resources should be deployed on the cluster by the vendor of the Kubernetes distribution, not the CSI driver vendor. These resources are not deployed on upstream Kubernetes 1.17.4, which is being used in this tutorial. Now, let’s deploy the necessary resources.\n",
    "\n",
    "**IMPORTANT:before, let´s check if we do have the right api version is already enabled, as in v1.17 we need v1beta1 as the endpoint for the snapshot.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:21:27.425017Z",
     "start_time": "2020-04-30T09:21:27.221745Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl api-versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if required you can exchange the create command to delete, to get the newest crd version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:21:58.728768Z",
     "start_time": "2020-04-30T09:21:53.737555Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#some resources may exist already\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**\n",
    "\n",
    "``as we are depolying not in the default namespace, we had to tweak the RBAC Clusterrolebinding for the serviceAccount (snapshot-controller).``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:41:25.506537Z",
     "start_time": "2020-04-30T12:41:25.307100Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role patched (no change)\n"
     ]
    }
   ],
   "source": [
    "kubectl patch ClusterRolebinding snapshot-controller-role -p '{\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"snapshot-controller\",\"namespace\":\"simple-demo\"}]}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:41:27.307298Z",
     "start_time": "2020-04-30T12:41:27.110760Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"apiVersion\": \"rbac.authorization.k8s.io/v1\",\n",
      "    \"kind\": \"ClusterRoleBinding\",\n",
      "    \"metadata\": {\n",
      "        \"annotations\": {\n",
      "            \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"ClusterRoleBinding\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"snapshot-controller-role\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"snapshot-controller-runner\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"snapshot-controller\\\",\\\"namespace\\\":\\\"default\\\"}]}\\n\"\n",
      "        },\n",
      "        \"creationTimestamp\": \"2020-04-27T16:40:58Z\",\n",
      "        \"name\": \"snapshot-controller-role\",\n",
      "        \"resourceVersion\": \"514147\",\n",
      "        \"selfLink\": \"/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/snapshot-controller-role\",\n",
      "        \"uid\": \"7d577154-0184-4820-b2e1-35b3bb5b6918\"\n",
      "    },\n",
      "    \"roleRef\": {\n",
      "        \"apiGroup\": \"rbac.authorization.k8s.io\",\n",
      "        \"kind\": \"ClusterRole\",\n",
      "        \"name\": \"snapshot-controller-runner\"\n",
      "    },\n",
      "    \"subjects\": [\n",
      "        {\n",
      "            \"kind\": \"ServiceAccount\",\n",
      "            \"name\": \"snapshot-controller\",\n",
      "            \"namespace\": \"simple-demo\"\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "kubectl get ClusterRoleBinding snapshot-controller-role -o json  # namespace is patched from default to simple-demo in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create the VolumeSnapshotClass\n",
    "\n",
    "<img src=\"pictures/snapshoting.png\" alt=\"SC\" width=\"550\" height=\"550\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each CSI driver that supports snapshots, at least one VolumeSnapshotClass object needs to be created. There’s only one backend that supports snapshots on this cluster and the VolumeSnapshotClass is therefore marked as default, which makes it easy for users to not care about implementation details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:41:54.405573Z",
     "start_time": "2020-04-30T12:41:53.409090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "volumesnapshotclass.snapshot.storage.k8s.io/hpe-snapshot created\n"
     ]
    }
   ],
   "source": [
    "kubectl apply -f - << EOF\n",
    "---\n",
    "apiVersion: snapshot.storage.k8s.io/v1beta1 \n",
    "kind: VolumeSnapshotClass\n",
    "metadata:\n",
    "  name: hpe-snapshot\n",
    "  annotations:\n",
    "    snapshot.storage.kubernetes.io/is-default-class: \"true\"\n",
    "driver: csi.hpe.com\n",
    "deletionPolicy: Delete\n",
    "parameters:\n",
    "  description: \"Snapshot created by the HPE CSI Driver\"\n",
    "  csi.storage.k8s.io/snapshotter-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:41:58.312573Z",
     "start_time": "2020-04-30T12:41:57.982626Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME           DRIVER        DELETIONPOLICY   AGE\n",
      "hpe-snapshot   csi.hpe.com   Delete           4s\n",
      "apiVersion: v1\n",
      "items:\n",
      "- apiVersion: snapshot.storage.k8s.io/v1beta1\n",
      "  deletionPolicy: Delete\n",
      "  driver: csi.hpe.com\n",
      "  kind: VolumeSnapshotClass\n",
      "  metadata:\n",
      "    annotations:\n",
      "      kubectl.kubernetes.io/last-applied-configuration: |\n",
      "        {\"apiVersion\":\"snapshot.storage.k8s.io/v1beta1\",\"deletionPolicy\":\"Delete\",\"driver\":\"csi.hpe.com\",\"kind\":\"VolumeSnapshotClass\",\"metadata\":{\"annotations\":{\"snapshot.storage.kubernetes.io/is-default-class\":\"true\"},\"name\":\"hpe-snapshot\"},\"parameters\":{\"csi.storage.k8s.io/snapshotter-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/snapshotter-secret-namespace\":\"kube-system\",\"description\":\"Snapshot created by the HPE CSI Driver\"}}\n",
      "      snapshot.storage.kubernetes.io/is-default-class: \"true\"\n",
      "    creationTimestamp: \"2020-04-30T12:41:54Z\"\n",
      "    generation: 1\n",
      "    name: hpe-snapshot\n",
      "    resourceVersion: \"541947\"\n",
      "    selfLink: /apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotclasses/hpe-snapshot\n",
      "    uid: e17068a6-1ab5-4ef6-b39a-e4c3d26ccbdc\n",
      "  parameters:\n",
      "    csi.storage.k8s.io/snapshotter-secret-name: nimble-secret\n",
      "    csi.storage.k8s.io/snapshotter-secret-namespace: kube-system\n",
      "    description: Snapshot created by the HPE CSI Driver\n",
      "kind: List\n",
      "metadata:\n",
      "  resourceVersion: \"\"\n",
      "  selfLink: \"\"\n"
     ]
    }
   ],
   "source": [
    "kubectl get VolumeSnapshotClass\n",
    "kubectl get VolumeSnapshotClass -o yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create Snapshot (User request)\n",
    "\n",
    "<img src=\"pictures/snapshot.png\" alt=\"SC\" width=\"950\" height=\"950\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:42:41.137415Z",
     "start_time": "2020-04-30T12:42:40.390989Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "volumesnapshot.snapshot.storage.k8s.io/ddpvc-clone-snap1 created\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "apiVersion: snapshot.storage.k8s.io/v1beta1\n",
    "kind: VolumeSnapshot\n",
    "metadata:\n",
    "  name: ddpvc-clone-snap1\n",
    "spec:\n",
    "  source:\n",
    "    persistentVolumeClaimName: \"ddpvc1-clone\"\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:42:48.142527Z",
     "start_time": "2020-04-30T12:42:47.953565Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT   CREATIONTIME   AGE\n",
      "ddpvc-clone-snap1                ddpvc1-clone                                                                                          7s\n"
     ]
    }
   ],
   "source": [
    "kubectl get volumesnapshot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-28T19:08:25.858518Z",
     "start_time": "2020-04-28T19:08:25.482018Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl delete volumesnapshot ddpvc-clone-snap1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## expand an Volume\n",
    "\n",
    "<img src=\"pictures/expand.png\" alt=\"SC\" width=\"350\" height=\"350\" style=\"float:left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform expansion operations on Kubernetes 1.14+, you must enhance your StorageClass with some additional attributes.\n",
    "\n",
    "``kubectl patch sc ddnimble -p '{ \"allowVolumeExpansion\": true }'``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:43:25.046060Z",
     "start_time": "2020-04-30T12:43:24.849144Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:            ddnimble\n",
      "IsDefaultClass:  No\n",
      "Annotations:     kubectl.kubernetes.io/last-applied-configuration={\"allowVolumeExpansion\":true,\"apiVersion\":\"storage.k8s.io/v1\",\"kind\":\"StorageClass\",\"metadata\":{\"annotations\":{},\"name\":\"ddnimble\"},\"parameters\":{\"accessProtocol\":\"iscsi\",\"allowOverrides\":\"description, limitIops, limitMbps, folder, destroyOnDelete\",\"csi.storage.k8s.io/controller-expand-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/controller-expand-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/controller-publish-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/controller-publish-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/fstype\":\"xfs\",\"csi.storage.k8s.io/node-publish-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/node-publish-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/node-stage-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/node-stage-secret-namespace\":\"kube-system\",\"csi.storage.k8s.io/provisioner-secret-name\":\"nimble-secret\",\"csi.storage.k8s.io/provisioner-secret-namespace\":\"kube-system\",\"description\":\"Volume created by the HPE CSI Driver for Kubernetes\",\"destroyOnDelete\":\"true\",\"folder\":\"simple-demo\",\"limitIops\":\"38400\",\"limitMbps\":\"2048\",\"reclaimPolicy\":\"Delete\",\"volumeBindingMode\":\"Immediate\"},\"provisioner\":\"csi.hpe.com\"}\n",
      "\n",
      "Provisioner:           csi.hpe.com\n",
      "Parameters:            accessProtocol=iscsi,allowOverrides=description, limitIops, limitMbps, folder, destroyOnDelete,csi.storage.k8s.io/controller-expand-secret-name=nimble-secret,csi.storage.k8s.io/controller-expand-secret-namespace=kube-system,csi.storage.k8s.io/controller-publish-secret-name=nimble-secret,csi.storage.k8s.io/controller-publish-secret-namespace=kube-system,csi.storage.k8s.io/fstype=xfs,csi.storage.k8s.io/node-publish-secret-name=nimble-secret,csi.storage.k8s.io/node-publish-secret-namespace=kube-system,csi.storage.k8s.io/node-stage-secret-name=nimble-secret,csi.storage.k8s.io/node-stage-secret-namespace=kube-system,csi.storage.k8s.io/provisioner-secret-name=nimble-secret,csi.storage.k8s.io/provisioner-secret-namespace=kube-system,description=Volume created by the HPE CSI Driver for Kubernetes,destroyOnDelete=true,folder=simple-demo,limitIops=38400,limitMbps=2048,reclaimPolicy=Delete,volumeBindingMode=Immediate\n",
      "AllowVolumeExpansion:  True\n",
      "MountOptions:          <none>\n",
      "ReclaimPolicy:         Delete\n",
      "VolumeBindingMode:     Immediate\n",
      "Events:                <none>\n"
     ]
    }
   ],
   "source": [
    "kubectl describe sc ddnimble "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:43:34.208058Z",
     "start_time": "2020-04-30T12:43:33.898351Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filesystem          Size  Used Avail Use% Mounted on\n",
      "/dev/mapper/mpathw  5.0G   33M  5.0G   1% /export\n"
     ]
    }
   ],
   "source": [
    "kubectl exec -it ddpod2 -- bash -c 'df -h /export'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:44:11.471245Z",
     "start_time": "2020-04-30T12:44:10.325942Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "persistentvolumeclaim/ddpvc1 configured\n"
     ]
    }
   ],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "apiVersion: v1\n",
    "kind: PersistentVolumeClaim\n",
    "metadata:\n",
    "  name: ddpvc1\n",
    "  annotations:\n",
    "    csi.hpe.com/description: \"demo the nimble plugin for K8S/Openshift/Docker/Rancher\"\n",
    "    csi.hpe.com/limitIOPS: \"8000\"\n",
    "    csi.hpe.com/destroyOnDelete: \"true\" \n",
    "spec:\n",
    "  accessModes:\n",
    "  - ReadWriteOnce\n",
    "  resources:\n",
    "    requests:\n",
    "      storage: 16Gi\n",
    "  storageClassName: ddnimble  \n",
    "EOF\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:44:41.801639Z",
     "start_time": "2020-04-30T12:44:41.461506Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filesystem          Size  Used Avail Use% Mounted on\n",
      "/dev/mapper/mpathw  5.0G   33M  5.0G   1% /export\n"
     ]
    }
   ],
   "source": [
    "# IMPORTANT: it may take a while until the new size is applied (est.: 1 Minute)\n",
    "kubectl exec -it ddpod2 -- bash -c 'df -h /export'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-29T15:30:32.505741Z",
     "start_time": "2020-04-29T15:30:32.317257Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl get pv,pvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## import an pre-existing Volume \n",
    "(Volume exists on Nimble, but unknown to k8s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:31:04.656992Z",
     "start_time": "2020-04-30T09:31:02.797310Z"
    }
   },
   "outputs": [],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "apiVersion: storage.k8s.io/v1\n",
    "kind: StorageClass\n",
    "metadata:\n",
    "  name: import-nimble-volume\n",
    "provisioner: csi.hpe.com\n",
    "parameters:\n",
    "  description: \"Volume provisioned by the HPE CSI Driver\"\n",
    "  accessProtocol: \"iscsi\"\n",
    "  csi.storage.k8s.io/fstype: xfs\n",
    "  csi.storage.k8s.io/provisioner-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/provisioner-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/controller-publish-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/node-stage-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/node-stage-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/node-publish-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/node-publish-secret-namespace: kube-system\n",
    "  csi.storage.k8s.io/controller-expand-secret-name: nimble-secret\n",
    "  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system\n",
    "  destroyOnDelete: \"true\"\n",
    "  # importVolAsClone: \"VolumetoImport\"\n",
    "  # importVolumeName: \"VolumetoImport\"\n",
    "# Extras...|\n",
    "  allowOverrides: limitIops, limitMbps, folder, importVolAsClone, importVolumeName\n",
    "#allowVolumeExpansion: True\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:31:12.745536Z",
     "start_time": "2020-04-30T09:31:12.558118Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl get sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "important:\n",
    "**allowOverrides: limitIops, limitMbps, folder, importVolAsClone, importVolumeName**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``Import note: if you use the importVolumeName, the name of the volume will be renamed\n",
    "form myDatabase to pvc-<uid>``\n",
    "\n",
    "``myimportedvolume   Bound    pvc-66b42c29-dcdc-413e-8a28-c147be4aa1d9   5Gi        RWO            import-nimble-volume   6s``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:32:00.520270Z",
     "start_time": "2020-04-30T09:31:59.271677Z"
    }
   },
   "outputs": [],
   "source": [
    "cat << 'EOF' | kubectl apply -f -\n",
    "---\n",
    "kind: PersistentVolumeClaim\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: myimportedvolume\n",
    "  annotations:  \n",
    "    csi.hpe.com/importVolAsClone: \"myDatabase\"\n",
    "    csi.hpe.com/description: \"myDatabase Demo Volume\"\n",
    "    # csi.hpe.com/importVolumeName: \"myDatabase\"\n",
    "    csi.hpe.com/folder: \"simple-demo\"\n",
    "spec:\n",
    "  accessModes:\n",
    "    - ReadWriteOnce\n",
    "  resources:\n",
    "    requests:\n",
    "      storage: 5Gi\n",
    "  storageClassName: import-nimble-volume\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:32:03.686380Z",
     "start_time": "2020-04-30T09:32:03.500280Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "kubectl get pvc "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:32:16.358815Z",
     "start_time": "2020-04-30T09:32:15.131100Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl create -f - << EOF\n",
    "---\n",
    "kind: Pod\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: ddpod4\n",
    "spec:\n",
    "  containers:\n",
    "  - name: nginx\n",
    "    image: nginx\n",
    "    volumeMounts:\n",
    "    - name: export\n",
    "      mountPath: /export\n",
    "  restartPolicy: Always\n",
    "  volumes:\n",
    "  - name: export\n",
    "    persistentVolumeClaim:\n",
    "      claimName: myimportedvolume\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:33:03.484436Z",
     "start_time": "2020-04-30T09:33:03.302202Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T09:33:07.334222Z",
     "start_time": "2020-04-30T09:33:07.038196Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl exec -it ddpod4 ls /export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:57:03.181384Z",
     "start_time": "2020-04-30T12:56:51.701467Z"
    }
   },
   "outputs": [],
   "source": [
    "kubectl delete pod ddpod2 ddpod3 ddpod4 snapshot-controller-0\n",
    "kubectl delete sc ddnimble\n",
    "kubectl delete sc import-nimble-volume\n",
    "kubectl delete volumesnapshot ddpvc-clone-snap1\n",
    "kubectl delete volumesnapshotclass hpe-snapshot\n",
    "kubectl delete ns simple-demo\n",
    "#kubectl delete pv $(kubectl get pv| grep -i simple-demo| awk ' { print $1 }')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-30T12:57:46.069516Z",
     "start_time": "2020-04-30T12:57:45.882592Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                   STATUS   AGE\n",
      "default                Active   2d21h\n",
      "hpecp                  Active   2d20h\n",
      "kd-apps                Active   2d20h\n",
      "kube-node-lease        Active   2d21h\n",
      "kube-public            Active   2d21h\n",
      "kube-system            Active   2d21h\n",
      "kubernetes-dashboard   Active   2d20h\n",
      "mapr-csi               Active   2d20h\n"
     ]
    }
   ],
   "source": [
    "kubectl get namespaces"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "307.188px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}