From e3e94b180fc3cd94ed0a7f993109d06994fa44c1 Mon Sep 17 00:00:00 2001 From: Christian Thiel Date: Fri, 17 Jan 2025 08:14:56 +0100 Subject: [PATCH] Add RDBMS backup documentation --- .../conceptual-differences.md | 8 +- .../elasticsearch-privileges.md | 0 .../elasticsearch}/opensearch-privileges.md | 0 .../concepts/databases/overview.md | 44 ++++++++ .../databases/relational-db/configuration.md | 104 ++++++++++++++++++ .../backup-restore/backup-and-restore.md | 32 +++++- sidebars.js | 18 ++- 7 files changed, 202 insertions(+), 4 deletions(-) rename docs/self-managed/concepts/{ => databases/elasticsearch}/elasticsearch-privileges.md (100%) rename docs/self-managed/concepts/{ => databases/elasticsearch}/opensearch-privileges.md (100%) create mode 100644 docs/self-managed/concepts/databases/overview.md create mode 100644 docs/self-managed/concepts/databases/relational-db/configuration.md diff --git a/docs/guides/migrating-from-camunda-7/conceptual-differences.md b/docs/guides/migrating-from-camunda-7/conceptual-differences.md index 02dde3361b9..73c8f3b3ccc 100644 --- a/docs/guides/migrating-from-camunda-7/conceptual-differences.md +++ b/docs/guides/migrating-from-camunda-7/conceptual-differences.md @@ -120,9 +120,15 @@ With Camunda 8 you deploy: - Your Spring Boot application with all custom code and the Zeebe client embedded. This application is typically scaled to at least two instances (for resilience) - The Zeebe Broker, typically scaled to at least three instances (for resilience) -- An elastic database (for Operate, Tasklist, and Optimize) +- A database (for Operate, Tasklist and the C8 REST API). Here you can choose between: + - Relational database. Currently supported are: H2, PostgreSQL, MariaDB, and Oracle. + - Elasticsearch +- An elastic database (for Operate) - Optimize, Operate, and Tasklist (each one is a Java application). You can scale those applications to increase availability if you want. +:::note +TODO: Adjust Image to RDBMS and Elasticsearch +::: ![Camunda 7 vs Camunda 8 Deployment View](../img/camunda7-vs-camunda8-deployment-view.png) Camunda 8 deployments happen within Kubernetes. There are [Helm charts available](self-managed/setup/overview.md) if you want to run Camunda 8 Self-Managed. diff --git a/docs/self-managed/concepts/elasticsearch-privileges.md b/docs/self-managed/concepts/databases/elasticsearch/elasticsearch-privileges.md similarity index 100% rename from docs/self-managed/concepts/elasticsearch-privileges.md rename to docs/self-managed/concepts/databases/elasticsearch/elasticsearch-privileges.md diff --git a/docs/self-managed/concepts/opensearch-privileges.md b/docs/self-managed/concepts/databases/elasticsearch/opensearch-privileges.md similarity index 100% rename from docs/self-managed/concepts/opensearch-privileges.md rename to docs/self-managed/concepts/databases/elasticsearch/opensearch-privileges.md diff --git a/docs/self-managed/concepts/databases/overview.md b/docs/self-managed/concepts/databases/overview.md new file mode 100644 index 00000000000..34e304f5c3a --- /dev/null +++ b/docs/self-managed/concepts/databases/overview.md @@ -0,0 +1,44 @@ +--- +id: overview +title: "Overview" +description: "An overview about possible databases" +--- + +To service applications like Operate, Tasklist and the Camunda REST API with data, Camunda 8 requires a database. This database stores data for all processes, processInstances, decisions, tasks, users etc. + +Camunda 8 supports the following databases: + +## Document based databases + +TODO: Add more text about behavior and limitations? + +Document based databases like ElasticSearch and OpenSearch + +## Relational databases + +TODO: Decide which minimal version is supported + +Currently, the following relational databases are officially supported: + +- PostgreSQL (17+) +- Oracle (23+) +- MariaDB (11.6+) +- H2 (2.3) + +To use and configure an RDBMS as storage backend, follow the instructions in the [Configuration](./relational-db/configuration.md) section. + +### Limitations + +When using a relational database as storage backend, the following limitations apply: + +#### Variables + +[The Camunda REST API](./../../../../apis-tools/camunda-api-rest/camunda-api-rest-overview) allows to fetch and filter for [Variables](./../../../../apis-tools/camunda-api-rest/specifications/get-variable). +To keep the API performant, the following comparison operations are only applied on the first 4000 characters of a String or JSON variable value: + +- equals +- notEquals +- in +- notIn + +The LIKE operator is not affected by this limitation. diff --git a/docs/self-managed/concepts/databases/relational-db/configuration.md b/docs/self-managed/concepts/databases/relational-db/configuration.md new file mode 100644 index 00000000000..71288169aaf --- /dev/null +++ b/docs/self-managed/concepts/databases/relational-db/configuration.md @@ -0,0 +1,104 @@ +--- +id: database-configuration +title: "Configuration" +--- + +## Installation + +To enable Camunda to use a relational database, two main configuration steps are required: + +1. Add the RDBMS [Exporter](./../../exporters.md) to the Zeebe broker configuration. +2. Enable the RDBMS as primary database for [Camunda REST API](./../../../../apis-tools/camunda-api-rest/camunda-api-rest-overview) +3. Configure the database connection. This is done by configuring the Spring datasource. + +```yaml +# enable the exporter +zeebe: + broker: + exporters: + rdbms: + className: RdbmsExporter + args: + # see Configuration section for configuration options + +# set the RDBMS as primary database for Camunda REST API +camunda: + database: + type: rdbms + url: jdbc:postgresql:camunda # example for a local PostgreSQL database with schema "camunda" + username: camunda + password: camunda +``` + +The RDBMS [Exporter](./../../exporters.md) configuration can be enabled in addition to other [Exporter](./../../exporters.md) as well. Note, that adding more exporters, can slow down the cluster. + +#### DB Schema management + +By default, Camunda will automatically create and update the database schema on startup. For this it uses the tool [Liquibase](https://www.liquibase.com/). In addition to the Camunda core tables, Liquibase will create the two tables `DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`. These tables must not be changed or updated. + +If the target database schema is not empty, it sometimes can be useful to define a prefix for the Camunda tables. This can be done by setting the following property in the configuration: + +```yaml +camunda.database.index-prefix: c8_ +``` + +To disable the automatic schema management, set the following property in the configuration: + +```yaml +camunda.database.auto-ddl: false +``` + +Native SQL scripts for all supported databases can be found in the Camunda distribution (TODO:LINK). These scripts can be used to create the schema manually. + +### Database privileges + +To use Camunda with the RDBMS, the configured database user must have the following privileges on the all Camunda tables (the actual privilege names might depend on the database vendor): + +- SELECT +- INSERT +- UPDATE +- DELETE + +Optionally, to enable the automatic schema management by Camunda and [Liquibase](https://www.liquibase.com/), the additional privileges must have been granted before the first start of any Camunda instance: + +- CREATE +- ALTER +- DROP + +Optionally, to enable the PURGE (TODO: Link) feature of the RDBMS Exporter, the following privilege must be granted to the camunda database user on all Camunda tables: + +- TRUNCATE + +## Configuration + +### Database configuration + +Camunda RDBMS database configuration reference: + +| Property name | Description | Default setting | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- | +| camunda.database.url | The JDBC connection url of the database | _empty_ | +| camunda.database.user | The username for the database connection | _empty_ | +| camunda.database.password | The password for the database connection | _empty_ | +| camunda.database.auto-ddl | If the Liquibase schemamanagement should be used or not. If not, the DBA has to install the schema from available [scripts](#db-schema-management) | true | +| camunda.database.index-prefix | A custom prefix for all camunda related database objects. | '' (empty) | +| camunda.database.database-vendor-id | Camunda uses vendor auto-detection vendor specific functions. With this property this auto-detection can be overridden. Possible values: _h2_, _mariadb_, _oracle_, _postgres_ | _empty_ | + +### Exporter Configuration + +The RDBMS Exporter can be configured in the `args` section of the exporter configuration: + +```yaml +# enable the exporter +zeebe.broker.exporters: + rdbms: + args: + propertyName: propertyValue +``` + +The following configuration options are available: + +| Property name | Description | Default setting | +| ------------- | --------------------------------------------------------------------------------------------------------------------- | --------------- | +| flushInterval | The maximum time an exported record waits in the flush queue before being flushed and committed to the database | 500 (ms) | +| maxQueueSize | The maximum number of records that can be added to the flush queue before being flushed and committed to the database | 1000 | diff --git a/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md b/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md index c5d68a21004..70a8d598e47 100644 --- a/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md +++ b/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md @@ -24,6 +24,8 @@ Zeebe stores its backup to an external storage and must be configured before the ### Backup process +#### Backups using ElasticSearch + The backup of each component and the backup of a Camunda 8 cluster is identified by an id. This means a backup `x` of Camunda 8 consists of backup `x` of Zeebe, backup `x` of Optimize, backup `x` of Operate, and backup `x` of Tasklist. The backup ID must be an integer and greater than the previous backups. :::note @@ -62,9 +64,29 @@ By default, the indices are prefixed with `zeebe-record`. If you have configured If any of the steps above fail, you may have to restart with a new backup id. Ensure exporting is resumed if the backup process force quits in the middle of the process. ::: +#### Backups using an RDBMS + +When using an RDBMS like PostgreSQL, Oracle or MariaDB, the backups of Zeebe and the database have to be taken separately and also depend on the database system. + +The backup of Zeebe is identified by an id `x`. The backup ID must be an integer and greater than the previous backups. + +:::note +We recommend using the timestamp as the backup id. +::: + +To back up a Camunda cluster using an RDBMS, follow these steps: + +1. Soft pause exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). +2. Take a backup of the relational database. See the documentation of the database system you are using. It is recommended to identify the database backup with the same backupId `x` as will be used for the Zeebe backup. +3. Take a backup `x` of Zeebe. See [how to take a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). +4. Wait until the backup `x` of Zeebe is completed before proceeding. See [how to monitor a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). + Resume exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). + ### Restore -To restore a Camunda 8 cluster from a backup, all components must be restored from their backup corresponding to the same backup id: +#### Restoring using ElasticSearch + +To restore an ElasticSearch based Camunda 8 cluster from a backup, all components must be restored from their backup corresponding to the same backup id: 1. Start Zeebe, Operate, Tasklist, and Optimize. (To ensure templates/aliases etc. are created) 2. Confirm proper configuration (such as shards, replicas count, etc.) @@ -74,3 +96,11 @@ To restore a Camunda 8 cluster from a backup, all components must be restored fr 6. Restore `zeebe-records*` indices from Elasticsearch snapshot. 7. Restore [Zeebe](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). 8. Start Zeebe, Operate, Tasklist, and Optimize. + +#### Restoring using an RDBMS + +To restore an RDBMS based Camunda 8 cluster from a backup, follow these steps: + +1. Restore the relational database backup into an empty database. +2. Restore [Zeebe](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). +3. Start Zeebe, Operate, Tasklist, and Optimize. diff --git a/sidebars.js b/sidebars.js index 07eb53e7495..ce28902497c 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1085,10 +1085,24 @@ module.exports = { { "Multi-region": ["self-managed/concepts/multi-region/dual-region"], }, + { + Databases: [ + "self-managed/concepts/databases/overview", + { + Elasticsearch: [ + "self-managed/concepts/databases/elasticsearch/elasticsearch-privileges", + "self-managed/concepts/databases/elasticsearch/opensearch-privileges", + ], + }, + { + "Relational databases": [ + "self-managed/concepts/databases/relational-db/database-configuration", + ], + }, + ], + }, "self-managed/concepts/multi-tenancy", "self-managed/concepts/mapping-rules", - "self-managed/concepts/elasticsearch-privileges", - "self-managed/concepts/opensearch-privileges", ], }, {