Adding option to use an Azure Backup Vault (#2)

This PR introduces the option to enable cloud provider specific backup solutions. Since Azure is the only supported Cloud Platform so far, the implementation uses Azure Backup Vault for this.

README.md

@@ -102,6 +102,9 @@ backends:  # Configuration for the different backends. Required fields are only
       vnets:  # List of vnets the storage account should allow access from. Each vnet listed here must have Microsoft.Storage added to the ServiceEndpoints collection of the subnet, optional
         - vnet: foobar-vnet  # Name of the virtual network, required
           subnet: default  # Name of the subnet, required
+    backup: # Configuration for use of Azure Backup Services. vault_name and policy_id are mandatory, if you want to use Azure Backup
+      vault_name: foobar-vault  # The name of the existing backup vault, make sure the Storage Account has the Role Assignment "Storage Account Backup Contributor" for the according vault
+      policy_id: 123123123  # The policy within the backup vault to use
     parameters:  # Fields here define defaults for parameters also in the CRD and are used if the parameter is not set in the custom object supplied by the user
       network:
         public_access: false  # If set to true no network restrictions are placed on the storage account, if set to false access is only possible through vnet and firewall rules, optional
@@ -115,6 +118,9 @@ backends:  # Configuration for the different backends. Required fields are only
         days: 2  # Number of days to keep deleted data, optional
       sftp:  # SFTP feature can only be enabled for the first time at creation of the storage account. Background: The hierarchical namespace setting is needed for SFTP and will be used implicitly but it can be only set at creation time.
         enabled: false  # enable SFTP interface, optional
+      backup:
+        enabled: false  # If enabled, the storage accounts will be added to an existing backup vault by default. Backup instances will not be cleaned up with Object Storage Buckets for recovery purposes
+
 ```
 
 Single configuration options can also be provided via environment variables, the complete path is concatenated using underscores, written in uppercase and prefixed with `HYBRIDCLOUD_`. As an example: `backends.azureblob.subscription_id` becomes `HYBRIDCLOUD_BACKENDS_AZUREBLOB_SUBSCRIPTION_ID`.
@@ -124,6 +130,8 @@ The azure backend also support a feature called `fake deletion` (via options `de
 
 For the azureblob backend there are several ways to protect the storage accounts from external access. One is on the network layer by disabling network access to the accounts from outside the cluster (via the `parameters.network.public_access` and `parameters.network.firewall_rules` and `network.vnets`) and the other is on the access layer by disallowing anonymous access (via `allow_anonymous_access`, this only gives the users the right to configure anonymous access, unless a user specifically does that only authenticated access is possible).
 
+The azureblob backend supports backups using [Azure Backup Vaults](https://learn.microsoft.com/en-us/azure/backup/backup-vault-overview). To enable Azure backup, first set the two fields `backup.vault_name` (the existing backup vault to use) and `backup.policy_id` (the existing policy to use). Now you can either enable backups by default using the field `parameters.backup.enabled` or configure backup per manifest using the field `backup.enabled`. Note: the configuration in the manifest overrides the global operator configuration.
+
 For the operator to interact with Azure it needs credentials. For local testing it can pick up the token from the azure cli but for real deployments it needs a dedicated service principal. Supply the credentials for the service principal using the environment variables `AZURE_SUBSCRIPTION_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_CLIENT_SECRET` (if you deploy via the helm chart use the use `envSecret` value). Depending on the backend the operator requires the following azure permissions within the scope of the resource group it deploys to:
 
 * `Microsoft.Storage/*`
@@ -182,6 +190,8 @@ spec:
     deleteRetention:  # Settings related to delete retention, optional
       enabled: false  # Enable retention on delete, optional
       retentionPeriodInDays: 1  # Days to keep deleted data, optional
+  backup:
+    enabled: false  # Override the default backup strategy configured in the global operator config
   containers:  # Only relevant for azure, list of containers to create in the bucket, for azure at least one is required, containers not on the list will be removed from the storage account, including their data
     - name: assets  # Name of the container, required
       anonymousAccess: false  # If set to true objects in the container can be accessed without authentication/authorization, only relevant if `security.anonymousAccess` is set to true, optional
@@ -215,7 +225,7 @@ To run it locally follow these steps:
 1. Create and activate a local python virtualenv
 2. Install dependencies: `pip install -r requirements.txt`
 3. Setup a local kubernetes cluster, e.g. with k3d: `k3d cluster create`
-4. Apply the CRDs in your local cluster: `kubectl apply -f helm/hybrid-cloud-object-storage-operator/crds/`
+4. Apply the CRDs in your local cluster: `kubectl apply -f helm/hybrid-cloud-object-storage-operator-crds/templates/`
 5. If you want to deploy to azure: Either have the azure cli installed and configured with an active login or export the following environment variables: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`
 6. Adapt the `config.yaml` to suit your needs
 7. Run `kopf run main.py -A`

helm/hybrid-cloud-object-storage-operator-crds/templates/objectstorage.yaml

@@ -97,6 +97,11 @@ spec:
                           type: boolean
                         retentionPeriodInDays:
                           type: number
+                backup:
+                  type: object
+                  properties:
+                    enabled:
+                      type: boolean
                 containers:
                   type: array
                   items:

hybridcloud/backends/azureblob.py

@@ -6,9 +6,11 @@
     StorageAccountCheckNameAvailabilityParameters, StorageAccountRegenerateKeyParameters, \
     DeleteRetentionPolicy, RestorePolicyProperties, ChangeFeed, LocalUser, PermissionScope, SshPublicKey
 from azure.mgmt.resource.locks.models import ManagementLockObject
-from ..util.azure import azure_client_storage, azure_client_locks
+from azure.mgmt.dataprotection.models import BackupInstanceResource, BackupInstance, PolicyInfo, Datasource
+from ..util.azure import azure_client_storage, azure_client_locks, azure_backup_client
 from ..config import config_get
 from ..util.reconcile_helpers import field_from_spec
+from ..util.exceptions import DeletionWithBackupEnabledException
 
 TAGS_PREFIX = "hybridcloud-object-storage-operator"
 HTTP_METHODS = ["DELETE", "GET", "HEAD", "MERGE", "OPTIONS", "PATCH", "POST", "PUT"]
@@ -36,6 +38,7 @@ def __init__(self, logger):
         self._logger = logger
         self._storage_client = azure_client_storage()
         self._lock_client = azure_client_locks()
+        self._backup_client = azure_backup_client()
         self._subscription_id = _backend_config("subscription_id", fail_if_missing=True)
         self._location = _backend_config("location", fail_if_missing=True)
         self._resource_group = _backend_config("resource_group", fail_if_missing=True)
@@ -76,6 +79,21 @@ def bucket_spec_valid(self, namespace, name, spec):
         if sftp_enabled and versioning:
             return (False, "SFTP and Versioning options cannot be both enabled")
 
+        backup_enabled = field_from_spec(spec, "backup.enabled", default=_backend_config("parameters.backup.enabled", default=False))
+
+        if backup_enabled:
+            vault_name = _backend_config("backup.vault_name", default=None)
+            policy_name = _backend_config("backup.policy_name", default=None)
+
+            if vault_name is None or policy_name is None:
+                return (False, "Backup is requested for this bucket but has not been configured for this backend in the operator configuration")
+        else:
+            backup_lock = self._get_backup_lock(bucket_name)
+
+            # Check if backup was enabled before
+            if backup_lock is not None:
+                return (False, "Backup was disabled, but has been enabled before. Disable Azure Backup for the storage account manually before deletion.")
+
         return (True, "")
 
     def bucket_exists(self, namespace, name):
@@ -93,6 +111,8 @@ def create_or_update_bucket(self, namespace, name, spec):
         tags = _calc_tags(namespace, name)
         sftp_enabled = field_from_spec(spec, "sftp.enabled", default=_backend_config("parameters.sftp.enabled",
                                                                                      default=False))
+        backup_enabled = field_from_spec(spec, "backup.enabled", default=_backend_config("parameters.backup.enabled", default=False))
+
         try:
             storage_account = self._storage_client.storage_accounts.get_properties(self._resource_group, bucket_name)
         except:
@@ -184,6 +204,37 @@ def create_or_update_bucket(self, namespace, name, spec):
                 if existing_username not in users_from_spec:
                     self._storage_client.local_users.delete(self._resource_group, bucket_name, existing_username)
 
+        if backup_enabled:
+            storage_account = self._storage_client.storage_accounts.get_properties(self._resource_group, bucket_name)
+
+            vault_name = _backend_config("backup.vault_name", fail_if_missing=True)
+            policy_name = _backend_config("backup.policy_name", fail_if_missing=True)
+
+            policy_id = f"/subscriptions/{self._subscription_id}/resourceGroups/{self._resource_group}/providers/Microsoft.DataProtection/backupVaults/{vault_name}/backupPolicies/{policy_name}"
+
+            backup_properties = BackupInstanceResource(
+                properties=BackupInstance(
+                    policy_info=PolicyInfo(
+                        policy_id=policy_id
+                    ),
+                    data_source_info=Datasource(
+                        datasource_type="Microsoft.Storage/storageAccounts/blobServices",
+                        resource_id=storage_account.id,
+                        resource_name=storage_account.name,
+                        resource_type="Microsoft.Storage/storageAccounts",
+                        resource_location=self._location,
+                    ),
+                    object_type="BackupInstance",
+                )
+            )
+
+            self._backup_client.backup_instances.begin_create_or_update(
+                resource_group_name=self._resource_group,
+                vault_name=vault_name,
+                backup_instance_name=bucket_name,
+                parameters=backup_properties
+            ).result()
+
         # Credentials
         for key in self._storage_client.storage_accounts.list_keys(self._resource_group, bucket_name).keys:
             if key.key_name == "key1":
@@ -203,6 +254,11 @@ def delete_bucket(self, namespace, name):
             tags = _calc_tags(namespace, name, {"marked-for-deletion": "yes"})
             self._storage_client.storage_accounts.update(self._resource_group, bucket_name, parameters=StorageAccountUpdateParameters(tags=tags))
         else:
+            backup_lock = self._get_backup_lock(bucket_name)
+
+            if backup_lock is not None:
+                raise DeletionWithBackupEnabledException(f"Failed to delete storage account {bucket_name}. Disable Azure Backup for the storage account manually before deletion.")
+
             self._storage_client.storage_accounts.delete(self._resource_group, bucket_name)
 
     def reset_credentials(self, namespace, name):
@@ -240,6 +296,19 @@ def _map_network_rules(self, spec, public_access):
             default_action="Allow" if public_access else "Deny"
         )
 
+    def _get_backup_lock(self, bucket_name):
+        try:
+            return self._lock_client.management_locks.get_at_resource_level(
+                lock_name="AzureBackupLock-DoNotDelete",
+                resource_name=bucket_name,
+                resource_type="storageAccounts",
+                resource_provider_namespace="Microsoft.Storage",
+                resource_group_name=self._resource_group,
+                parent_resource_path=""
+                )
+        except:
+            return None
+
 
 def _map_cors_rules(cors):
     if not cors:

hybridcloud/config.py

@@ -9,7 +9,7 @@
 
 class ConfigurationException(Exception):
     def __init__(self, description):
-        super().__init(description)
+        super().__init__(description)
 
 
 def _get_config_value_from_env(key):

hybridcloud/handlers/bucket.py

@@ -4,6 +4,7 @@
 from ..util.reconcile_helpers import ignore_control_label_change, process_action_label
 from ..util import k8s
 from ..util.constants import BACKOFF
+from ..util.exceptions import DeletionWithBackupEnabledException
 
 
 if config_get("handler_on_resume", default=False):
@@ -59,7 +60,13 @@ def bucket_delete(spec, status, name, namespace, logger, **kwargs):
     backend = bucket_backend(backend_name, logger)
     if backend.bucket_exists(namespace, name):
         logger.info("Deleting bucket")
-        backend.delete_bucket(namespace, name)
+        try:
+            backend.delete_bucket(namespace, name)
+        except DeletionWithBackupEnabledException as e:
+            reason = str(e)
+            _status(name, namespace, status, "failed", f"Deletion failed: {reason}")
+            raise kopf.TemporaryError(reason)
+
     else:
         logger.info("Bucket does not exist. Not doing anything")
     k8s.delete_secret(namespace, spec["credentialsSecret"])

hybridcloud/util/azure.py

@@ -1,6 +1,7 @@
 from azure.identity import DefaultAzureCredential
 from azure.mgmt.storage import StorageManagementClient
 from azure.mgmt.resource import ManagementLockClient
+from azure.mgmt.dataprotection import DataProtectionClient
 from ..config import get_one_of
 
 
@@ -18,3 +19,7 @@ def azure_client_storage():
 
 def azure_client_locks():
     return ManagementLockClient(_credentials(), _subscription_id())
+
+
+def azure_backup_client():
+    return DataProtectionClient(_credentials(), _subscription_id())

hybridcloud/util/exceptions.py

@@ -0,0 +1,2 @@
+class DeletionWithBackupEnabledException(Exception):
+    pass

requirements.txt

@@ -3,4 +3,5 @@ kopf==1.35.2
 azure-identity==1.7.1
 azure-mgmt-resource==20.0.0
 azure-mgmt-storage==19.1.0
+azure-mgmt-dataprotection==1.0.0b2
 pyyaml==6.0