From db6436c3a39246aae80b44694947c151720b7256 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Tue, 21 May 2024 19:43:59 +0530 Subject: [PATCH 01/67] BigAnimal - Health Status doc Added first draft of Health Status doc as per [DOCS-469](https://enterprisedb.atlassian.net/browse/DOCS-469) & [DOCS-470](https://enterprisedb.atlassian.net/browse/DOCS-470) --- .../health_status.mdx | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx new file mode 100644 index 00000000000..6b605a7a961 --- /dev/null +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -0,0 +1,40 @@ +--- +title: "Health Status" +--- + +Health Status dashboard provides the status of the Postgres high-availability clusters. It provides status for both Primary/Standby and Distributed high-availability clusters. + +Health Status dashboard displays: +- A set of cluster-wide health indicators that helps to draw your attention immediately to the critical issues. +- A schematic view of all the nodes organized in a cluster distributed across regions displaying health and role(s) of each node. +- A replication status view displaying the status of each replication slot and the associated replication lags. + +## Viewing Health Status dashboard + +To view the **Health Status** dashboard from the BigAnimal portal: + +1. In the left navigation of BigAnimal Portal, go to **Clusters**. +2. Select any ready cluster. +3. On the cluster detail page, select the **Health Status** tab. + +The **Health Status** tab displays the dashboard with health status categorized in sections: +- [Viewing Health Status dashboard](#viewing-health-status-dashboard) + - [Global cluster health](#global-cluster-health) + - [Regional nodes health and roles](#regional-nodes-health-and-roles) + - [Replication status](#replication-status) + +### Global cluster health + +The global cluster health section displays the cluster-wide view including the following metrics: + +- **Raft Status** (PGD only)— It indicates whether the Raft consensus algorithm is running correctly in the cluster. It verifies that one node is elected as the global leader and the Raft roles such as RAFT_LEADER and RAFT_FOLLOWER are defined correctly across all the nodes. +- **Replication Slots Status** (PGD only)— It indicates whether all the replication slots are in streaming state or not. +- **Clock Skew**— It indicates whether the node's clock is in sync and doesn't exceed a threshold of 60 seconds. +- **Proxy Status** (PGD only)— It provides the number of PGD proxies up over the ones available.???? +- **Node Status**— It provides the number of nodes that are up and running. +- **Transaction rate**— It provides the total number of transactions including committed and rolled back transactions per second in the cluster. + + +### Regional nodes health and roles + +### Replication status \ No newline at end of file From c0573ccb4140478da7870ba11e01b8bceb5d4b9a Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 21 May 2024 14:29:51 -0400 Subject: [PATCH 02/67] Edits to Migration Toolkit PR5803 --- .../55/04_functionality_overview.mdx | 2 +- .../08_mtk_command_options.mdx | 34 +++++++++---------- .../migration_toolkit/55/11_questions.mdx | 21 ++++++------ .../55/mtk_rel_notes/mtk_5580_rel_notes.mdx | 2 +- 4 files changed, 29 insertions(+), 30 deletions(-) diff --git a/product_docs/docs/migration_toolkit/55/04_functionality_overview.mdx b/product_docs/docs/migration_toolkit/55/04_functionality_overview.mdx index 1df7c31af7d..4e723fd2ae6 100644 --- a/product_docs/docs/migration_toolkit/55/04_functionality_overview.mdx +++ b/product_docs/docs/migration_toolkit/55/04_functionality_overview.mdx @@ -31,7 +31,7 @@ Migration Toolkit also allows you to migrate database objects and data to an EDB - On EDB Postgres Advanced Server, use the `-copyViaDBLinkOra` option to enable the `dblink_ora` module. - Use the `-connRetryCount ` option to specify the number of retry attempts to perform if the target database connection is lost during data migration. - Use the `-connRetryInterval ` option to specify the seconds to wait before each target database reconnection attempt during a data migration. -- Use the `-abortConnOnFailure ` option to specify if to abort the migration when all target database reconnection attempts during a data migration fail. +- Use the `-abortConnOnFailure ` option to specify whether to abort the migration when all target database reconnection attempts fail during a data migration. The default is `true`, which aborts the session if the connection fails after the the specified `-connRetryCount` threshold. - Use the `-pgIdleTxSessionTimeOut ` to override the value of the PostgreSQL or EDB Postgres Advanced Server `idle_in_transaction_session_timeout` configuration option in the MTK connection session. diff --git a/product_docs/docs/migration_toolkit/55/07_invoking_mtk/08_mtk_command_options.mdx b/product_docs/docs/migration_toolkit/55/07_invoking_mtk/08_mtk_command_options.mdx index d713c2936a1..fec55811d50 100644 --- a/product_docs/docs/migration_toolkit/55/07_invoking_mtk/08_mtk_command_options.mdx +++ b/product_docs/docs/migration_toolkit/55/07_invoking_mtk/08_mtk_command_options.mdx @@ -422,37 +422,37 @@ Specify each entry in the file on a separate line in a key=value pair. The left ## Connection retry options -Whenever there is a connection failure with the target database during a data migration, Migration Tookit automatically attempts to reconnect to the +Whenever there's a connection failure with the target database during a data migration, Migration Tookit attempts to reconnect to the target database to ensure the migration completes without skipping any tables. When the connection is reestablished, -Migration Toolkit restarts or resumes the data copy for the table that was being migrated when the connection was lost and then performs the data copy for the remaining tables. +Migration Toolkit restarts or resumes the data copy for the table that was being migrated when the connection was lost. It then performs the data copy for the remaining tables. Migration Toolkit resumes the migration based on the mode that it was using to migrate the data: - If Migration Toolkit was using Copy API to migrate the data, it continues copying data from the last failed row. Copy API is compatible with PostgreSQL and EDB Postgres Advanced Server as target databases. -- In other data migration modes, Migration Toolkit truncates the data on the target table and re-copies the entire failed table. +- In other data migration modes, Migration Toolkit truncates the data on the target table and recopies the entire failed table. !!!note Scope and limitations Database scope: The connection retry capability allows Migration Toolkit to reconnect to the target database. - Retry attempts for issues with the source database are not supported. + Retry attempts for issues with the source database aren't supported. Migration scope: This capability allows Migration Toolkit to retry migrating data. - Retry attempts for issues with the schema migration are not supported. + Retry attempts for issues with the schema migration aren't supported. Modality scope: This reconnection capability is available with the data migration mode (`-dataOnly`). - It is also available when you run a migration **without** specifying either the `-dataOnly` or `-schemaOnly` options. + It's also available when you run a migration without specifying either the `-dataOnly` or `-schemaOnly` options. !!! You can specify several connection retry options: `-connRetryCount []` -Use the `-connRetryCount` option to specify the number of retry attempts to perform in case the target database connection fails. -The `[]` value must be a number between 0 and 50, the default is 3 retry attempts. +Use the `-connRetryCount` option to specify the number of retry attempts to perform if the target database connection fails. +The `[]` value must be a number between 0 and 50. The default is 3 retry attempts. -Since the retry applies to the migration of data, it is not compatible with the `-schemaOnly` option. +Since the retry applies to the migration of data, it isn't compatible with the `-schemaOnly` option. Example: @@ -463,10 +463,10 @@ $ ./runMTK.sh -connRetryCount 2 -dataOnly -tables dept,emp,jobhist public `-connRetryInterval []` Use the `-connRetryInterval` option to specify the seconds to wait before each subsequent reconnection attempt -in case the target database connection fails. -The `[]` value must be a number between 0 and 900, the default is 30 seconds. +if the target database connection fails. +The `[]` value must be a number between 0 and 900. The default is 30 seconds. -Since the retry applies to the migration of data, it is not compatible with the `-schemaOnly` option. +Since the retry applies to the migration of data, it isn't compatible with the `-schemaOnly` option. Example: @@ -476,12 +476,12 @@ $ ./runMTK.sh -connRetryCount 2 -connRetryInterval 50 -dataOnly -tables dept,emp `-abortConnOnFailure` [true/false]` -Specify if to abort the migration if all the reconnection attempts failed. +Specify whether to abort the migration if all the reconnection attempts failed. -Set the `-abortConnOnFailure` to `false`, to skip migrating the failed table and proceed to the next table. -The default is `true`, which aborts the session if the connection fails after the the specified `-connRetryCount` threshold. +To skip migrating the failed table and proceed to the next table, set the `-abortConnOnFailure` to `false`. +The default is `true`, which aborts the session if the connection fails after the specified `-connRetryCount` threshold. -Since the retry applies to the migration of data, it is not compatible with the `-schemaOnly` option. +Since the retry applies to the migration of data, it isn't compatible with the `-schemaOnly` option. Example: @@ -493,7 +493,7 @@ $ ./runMTK.sh -connRetryCount 2 -connRetryInterval 50 -abortConnOnFailure false Specify the PostgreSQL or EDB Postgres Advanced Server `idle_in_transaction_session_timeout`, which defines the time after which the database terminates the session when a migration transaction is in idle state. -The `[]` value should be greater than 0, and the default is 180 seconds. +`[]` is a value greater than 0. The default is 180 seconds. Example: diff --git a/product_docs/docs/migration_toolkit/55/11_questions.mdx b/product_docs/docs/migration_toolkit/55/11_questions.mdx index 42bc8bdccc5..1d69c500c61 100644 --- a/product_docs/docs/migration_toolkit/55/11_questions.mdx +++ b/product_docs/docs/migration_toolkit/55/11_questions.mdx @@ -66,25 +66,24 @@ Migration scope: This capability allows Migration Toolkit to retry migrating dat Retry attempts for issues with the schema migration are currently not supported. Modality scope: This reconnection capability is available with the data migration mode (`-dataOnly`). -It is also available for the data migration step when you run a migration **without** specifying either the `-dataOnly` and `-schemaOnly` options. +It's also available for the data migration step when you run a migration without specifying either the `-dataOnly` and `-schemaOnly` options. -**What do I do if there is an error during a data migration that results in the data for one or more of my tables not being fully migrated?** +**What do I do if there's an error during a data migration that results in the data for one or more of my tables not being fully migrated?** -If Migration Toolkit is not able to reconnect successfully and one or more of the tables were not +If Migration Toolkit can't reconnect successfully and one or more of the tables weren't fully migrated, restart the entire data migration (with the `-dataOnly` option) or configure Migration Toolkit -to migrate only the tables that were not fully migrated in the previous run. Note that you can use the `connRetryCount`, +to migrate only the tables that weren't fully migrated in the previous run. You can use `connRetryCount`, `connRetryInterval`, and `abortConnOnFailure` to alter the retry configuration options. If the source database was accepting write transactions while the previous Migration Toolkit data migration was in process, -the full (all tables) data migration should be performed again to ensure that data migrated to the destination database is in a +perform the full (all tables) data migration again to ensure that data migrated to the destination database is in a consistent state. -If it can be confirmed that no write transactions were performed on the source database while the previous migration was being performed, -then it should be safe to migrate only those tables that had not been fully migrated. +If you can confirm that no write transactions were performed on the source database while the previous migration was being performed, +then it's safe to migrate only those tables that weren't fully migrated. -**What do I do if there is an error during a schema migration that results in not all of my schema objects being fully migrated?** +**What do I do if there's an error during a schema migration that results in not all of my schema objects being fully migrated?** -Unfortunately, reconnection for schema migration errors is not supported at this time. If Migration Toolkit is not able to migrate all +Unfortunately, reconnection for schema migration errors isn't supported at this time. If Migration Toolkit can't migrate all schemas, restart the entire schema migration again (with the `-schemaOnly` option), or configure Migration Toolkit to migrate only -the schemas that were not fully migrated in the previous run. - +the schemas that weren't fully migrated in the previous run. diff --git a/product_docs/docs/migration_toolkit/55/mtk_rel_notes/mtk_5580_rel_notes.mdx b/product_docs/docs/migration_toolkit/55/mtk_rel_notes/mtk_5580_rel_notes.mdx index f8b73946c50..1e367de9451 100644 --- a/product_docs/docs/migration_toolkit/55/mtk_rel_notes/mtk_5580_rel_notes.mdx +++ b/product_docs/docs/migration_toolkit/55/mtk_rel_notes/mtk_5580_rel_notes.mdx @@ -11,7 +11,7 @@ New features, enhancements, bug fixes, and other changes in Migration Toolkit 55 |--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| | Enhancement | Added the ability to automatically attempt to reconnect to the target database and resume or restart the data migration of the table that was being migrated when the connection was lost. | | | Enhancement | Migration Toolkit now allows the migration of stored `procedure` objects from a source PostgreSQL to a target PostgreSQL database. | | -| Enhancement | The exclude object capability (`-exclude` with a set of options) now supports the exclusion of subsets of `Sequences`, `Procedures`, `Functions`, `Packages`, `Synonyms`, and `Queues` from the migration. | #99271 / 31314 | +| Enhancement | The exclude object capability (`-exclude` with a set of options) now supports excluding the subsets of `Sequences`, `Procedures`, `Functions`, `Packages`, `Synonyms`, and `Queues` from the migration. | #99271 / 31314 | | Enhancement | The connection sessions initiated by Migration Toolkit are now identifiable with an application name. This facilitates the debugging of issues relevant to a source or target database. | | | Enhancement | Migration Toolkit now logs the names of missing tables as part of the error message when one or more tables are missing from the source database. | #98134 / 30362 | | Bug fix | Fixed an issue that caused migration failures for Oracle tables containing a numeric column with an empty default clause. | #100987 / 32874 | From 6f68b0acf8097ad4915a7113a1ba648b36ab5400 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 21 May 2024 14:59:42 -0400 Subject: [PATCH 03/67] Edits to EPRS PR5607 --- ...ictions_on_replicated_database_objects.mdx | 8 ++--- .../single_quote_unescape.mdx | 12 ++++---- .../skip_default_column.mdx | 8 ++--- .../snapshot_connection_retry.mdx | 29 +++++++++---------- .../7/eprs_rel_notes/eprs_rel_notes_7.8.0.mdx | 10 +++---- 5 files changed, 33 insertions(+), 34 deletions(-) diff --git a/product_docs/docs/eprs/7/02_overview/04_design_replication_system/03_restrictions_on_replicated_database_objects.mdx b/product_docs/docs/eprs/7/02_overview/04_design_replication_system/03_restrictions_on_replicated_database_objects.mdx index 43423bf47c8..7276d1d2392 100644 --- a/product_docs/docs/eprs/7/02_overview/04_design_replication_system/03_restrictions_on_replicated_database_objects.mdx +++ b/product_docs/docs/eprs/7/02_overview/04_design_replication_system/03_restrictions_on_replicated_database_objects.mdx @@ -37,7 +37,7 @@ The following are the restrictions on Oracle database objects: - You can use Oracle tables with the `RAW` data type in snapshot-only publications but not in synchronization replications. -- You can include Oracle materialised views in snapshot-only publications but not in synchronization replications. +- You can include Oracle materialized views in snapshot-only publications but not in synchronization replications. - You can't replicate Oracle tables that include the following data types: - `BFILE` @@ -179,7 +179,7 @@ HINT: Use DROP ... CASCADE to drop the dependent objects too. Since Replication Server drops the constraints using the `CASCADE` option, the above error would not occur; however, the result of using this option means that the view will also be dropped. !!! Important - When replicating to a target PGD instance, source `ALTER TABLE` statements will not be replicated due to a documented PGD limitation. - For example, if you use the `ALTER TABLE` command to change a column's type, this operation won't be replicated if the command causes the whole table to be rewritten. + When replicating to a target PGD instance, source `ALTER TABLE` statements aren't replicated due to a documented PGD limitation. + For example, if you use the `ALTER TABLE` command to change a column's type, this operation isn't replicated if the command causes the whole table to be rewritten. - For more details, see the [PGD ALTER TABLE documentation](https://www.enterprisedb.com/docs/pgd/latest/ddl/ddl-command-handling/#alter-table). + For more details, see the [PGD ALTER TABLE documentation](/pgd/latest/ddl/ddl-command-handling/#alter-table). diff --git a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx index 18cbb4d88cd..2f6e8371644 100644 --- a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx +++ b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx @@ -2,12 +2,12 @@ title: "Unescaping single-quote characters" --- -In the [Batch Update Using Prepared Statement (BUP)](/eprs/latest/05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) -mode, Replication Server adds an extra single quote character (') when it recognizes text data with single quotes during the wal-based replication. +In the [Batch Update Using Prepared Statement (BUP)](../../../05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) +mode, Replication Server adds an extra single quote character (') when it recognizes text data with single quotes during the WAL-based replication. This results in an additional single quote on the target database. -If there is a possibility that a source database table column contains a single quote character, -and you don't want the target database to contain an _additional_ single quote character, set the parameter `unescapeSingeQuote` to `true`. +If there's a possibility that a source database table column contains a single quote character, +and you don't want the target database to contain an additional single quote character, set the parameter `unescapeSingeQuote` to `true`. By default, this option is disabled (`unescapeSingeQuote=false`). @@ -15,8 +15,8 @@ By default, this option is disabled (`unescapeSingeQuote=false`). unescapeSingeQuote=true ``` -When set to `true`, the option removes the single quote character that was added during the wal-based replication operation in BUP mode. +When set to `true`, the option removes the single quote character that was added during the WAL-based replication operation in BUP mode. !!!note Note Enabling the `unescapeSingeQuote` option adds extra processing for each column, which can increase the replication time. - Therefore, enable this option only if there is a possibility that the source data contains single quotes and you configured Replication Server to use wal-based replication in BUP mode. \ No newline at end of file + Therefore, enable this option only if there's a possibility that the source data contains single quotes and you configured Replication Server to use WAL-based replication in BUP mode. \ No newline at end of file diff --git a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/skip_default_column.mdx b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/skip_default_column.mdx index e6b45e03e50..8db5265fcb5 100644 --- a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/skip_default_column.mdx +++ b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/skip_default_column.mdx @@ -4,17 +4,17 @@ title: "Skipping migration of column DEFAULT clause" ## Use cases -- The suscription database is in read-only mode. Therefore, DEFAULT clause columns serve no purpose as +- The subscription database is in read-only mode. Therefore, DEFAULT clause columns serve no purpose as they are already applied on the publication database. You can skip migrating DEFAULT columns for faster processing. -- Sequences are not migrated during replication operations. - If a DEFAULT clause depends on a Sequence, the DEFAULT clause migration will fail. +- Sequences aren't migrated during replication operations. + If a DEFAULT clause depends on a sequence, the DEFAULT clause migration will fail. You can skip migrating DEFAULT columns to avoid migration failures. ## Configuration -To skip the replication of a DEFAULT clause, update the below option to true: +To skip the replication of a DEFAULT clause, set the following option to true: ``` skipColDefaultClause=true diff --git a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/snapshot_connection_retry.mdx b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/snapshot_connection_retry.mdx index aa80acecc6c..44bfb8fc284 100644 --- a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/snapshot_connection_retry.mdx +++ b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/snapshot_connection_retry.mdx @@ -3,42 +3,41 @@ title: "Snapshot replication connection retry" --- When a connection failure with the subscription database occurs during a snapshot replication, -the Replication Server automatically attempts to reconnect to it. -The goal is that the snapshot replication completes without skipping any tables. When the connection is reestablished, -Replication Server restarts or resumes the snapshot replication procedure for the table that was being migrated when the connection was lost and then performs the data copy for the remaining tables. +Replication Server attempts to reconnect to it. +The goal is for the snapshot replication to complete without skipping any tables. When the connection is reestablished, +Replication Server restarts or resumes the snapshot replication procedure for the table that was being migrated when the connection was lost. It then performs the data copy for the remaining tables. !!!note Scope and limitations Database scope: The connection retry capability allows Replication Server to reconnect to the target/subscription database. - Retry attempts for connection issues with the source/publication database are not supported. + Retry attempts for connection issues with the source/publication database aren't supported. Replication scope: This capability allows Replication Server to retry replicating data. - Issues with schema replication are not supported. + Issues with schema replication aren't supported. !!! You can specify several connection retry options: - `-snapshotConnRetryCount []` -Use the `-snapshotConnRetryCount` option to specify the number of retry attempts to perform in case the subscription database connection fails in a Snapshot session. -The `[]` value must be a number between 0 and 50, the default is 3 retry attempts. +Use the `-snapshotConnRetryCount` option to specify the number of retry attempts to perform if the subscription database connection fails in a snapshot session. +The `[]` value must be a number between 0 and 50. The default is 3 retry attempts. - `-snapshotConnRetryInterval []` Use the `-snapshotConnRetryInterval` option to specify the seconds to wait before each subsequent reconnection attempt -in case the connection to the subscription database fails in a Snapshot session. -The `[]` value must be a number between 0 and 900, the default is 30 seconds. +if the connection to the subscription database fails in a snapshot session. +The `[]` value must be a number between 0 and 900. The default is 30 seconds. - `-snapshotAbortOnConnFailure [true/false]` -Specify if to abort the Snapshot replication if all the reconnection attempts fail. +Specify whether to abort the snapshot replication if all the reconnection attempts fail. -Set the `-snapshotAbortOnConnFailure` to `false`, to skip replicating the failed table and proceed to the next table. -The default is `true`, which aborts the Snapshot session if the connection fails after the specified `-snapshotConnRetryCount` threshold. +To skip replicating the failed table and proceed to the next table, set `-snapshotAbortOnConnFailure` to `false`. +The default is `true`, which aborts the snapshot session if the connection fails after the specified `-snapshotConnRetryCount` threshold. - `-snapshotPGIdleTxSessionTimeOut []` -Specify the `idle_in_transaction_session_timeout`, which defines the time after which +Specify `idle_in_transaction_session_timeout`, which defines the time after which the subscription database terminates the session when a snapshot replication transaction is in idle state. -The `[]` value should be greater than 0, and the default is 180 seconds. - +The `[]` value must be greater than 0. The default is 180 seconds. diff --git a/product_docs/docs/eprs/7/eprs_rel_notes/eprs_rel_notes_7.8.0.mdx b/product_docs/docs/eprs/7/eprs_rel_notes/eprs_rel_notes_7.8.0.mdx index 4b51db5c5bc..876925f05d8 100644 --- a/product_docs/docs/eprs/7/eprs_rel_notes/eprs_rel_notes_7.8.0.mdx +++ b/product_docs/docs/eprs/7/eprs_rel_notes/eprs_rel_notes_7.8.0.mdx @@ -13,14 +13,14 @@ New features, enhancements, bug fixes, and other changes in Replication Server 7 | Enhancement | Added the `skipColDefaultClause` subscription server configuration option to skip the column DEFAULT value for the subscription database. | #100987/32874 | | Enhancement | EDB Postgres Replication Server now provides packages and support for installations on SLES15 SP5. | | | Bug Fix | Fixed an issue where the synchronization cycle gets stuck for a case when a transaction is rolled back with SQL Server as the publication database. | #96964/29244 | -| Bug Fix | Fixed an issue where columns containing single quotes are not replicated properly for wal-based replication when Batch Update Using Prepared Statement mode is used. Introduced the `unescapeSingeQuote` option to control this processing. | #94921/27387, #95823/28200 | -| Bug Fix | Fixed an issue where some of the views' owners are changed after a snapshot. | #102472/34206 | +| Bug Fix | Fixed an issue where columns containing single quotes aren't replicated properly for WAl-based replication when Batch Update Using Prepared Statement mode is used. Introduced the `unescapeSingeQuote` option to control this processing. | #94921/27387, #95823/28200 | +| Bug Fix | Fixed an issue where some of the views' owners were changed after a snapshot. | #102472/34206 | | Bug Fix | Fixed an error observed during the removal of a SQL Server publication database. | | ## End-of-support notice -We recommend you update to the most recent software version. -*If you have not yet updated to the most current version, please see the end-of-support notes below*: +We recommend that you update to the most recent software version. +*If you haven't yet updated to the most current version, see the end-of-support notes that follow*: **Software:** Replication Server @@ -31,4 +31,4 @@ We recommend you update to the most recent software version. Additional details can be found at [EDB Platform Compatibility](https://www.enterprisedb.com/resources/platform-compatibility). !!! Note - Version 7.x provides a non-breaking upgrade path for existing 6.2.x-based cluster deployments; however, we strongly recommend that the upgrade be verified in a staging or non-production environment before applying the upgrade in a production environment. \ No newline at end of file + Version 7.x provides a non-breaking upgrade path for existing 6.2.x-based cluster deployments. However, we strongly recommend that the upgrade be verified in a staging or non-production environment before applying the upgrade in a production environment. \ No newline at end of file From 8b82eb38ad88c937de6913edd343363c1a6112ae Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 21 May 2024 15:01:04 -0400 Subject: [PATCH 04/67] Update product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx --- .../single_quote_unescape.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx index 2f6e8371644..15b48a1be27 100644 --- a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx +++ b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx @@ -2,7 +2,7 @@ title: "Unescaping single-quote characters" --- -In the [Batch Update Using Prepared Statement (BUP)](../../../05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) +In [Batch Update Using Prepared Statement (BUP)](../../../05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) mode, Replication Server adds an extra single quote character (') when it recognizes text data with single quotes during the WAL-based replication. This results in an additional single quote on the target database. From e0989db9145a60d1cc42859e6d12f780a0a74c63 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 21 May 2024 15:40:51 -0400 Subject: [PATCH 05/67] Update product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx --- .../single_quote_unescape.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx index 15b48a1be27..d5f1824c860 100644 --- a/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx +++ b/product_docs/docs/eprs/7/10_appendix/03_miscellaneous_xdb_processing_topics/01_publications_and_subscriptions_server_conf_options/single_quote_unescape.mdx @@ -2,7 +2,7 @@ title: "Unescaping single-quote characters" --- -In [Batch Update Using Prepared Statement (BUP)](../../../05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) +In [Batch Update Using Prepared Statement (BUP)](/eprs/latest/05_smr_operation/08_optimizing_performance/02_optimize_sync_replication/01_using_sql_statements/) mode, Replication Server adds an extra single quote character (') when it recognizes text data with single quotes during the WAL-based replication. This results in an additional single quote on the target database. From e9d371f163f25b3b453e61a54f0e2a4940e903c9 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Wed, 22 May 2024 15:40:07 +0530 Subject: [PATCH 06/67] Added more content --- .../health_status.mdx | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 6b605a7a961..0ea28a39b9a 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -28,13 +28,44 @@ The **Health Status** tab displays the dashboard with health status categorized The global cluster health section displays the cluster-wide view including the following metrics: - **Raft Status** (PGD only)— It indicates whether the Raft consensus algorithm is running correctly in the cluster. It verifies that one node is elected as the global leader and the Raft roles such as RAFT_LEADER and RAFT_FOLLOWER are defined correctly across all the nodes. -- **Replication Slots Status** (PGD only)— It indicates whether all the replication slots are in streaming state or not. +- **Replication Slot Status** (PGD only)— It indicates whether all the replication slots are in streaming state or not. - **Clock Skew**— It indicates whether the node's clock is in sync and doesn't exceed a threshold of 60 seconds. -- **Proxy Status** (PGD only)— It provides the number of PGD proxies up over the ones available.???? +- **Proxy Status** (PGD only)— It provides the number of PGD proxies up and running as compared to the available proxies. - **Node Status**— It provides the number of nodes that are up and running. - **Transaction rate**— It provides the total number of transactions including committed and rolled back transactions per second in the cluster. ### Regional nodes health and roles -### Replication status \ No newline at end of file +The regional nodes health and roles section displays fine-grained health status at regional and node level. It is structured as an accordion, with each element representing a group of nodes deployed in the same region. Each item displays basic information like: +- **Proxy Status** (only PGD)— It indicates the number of active proxies compared to the available proxies in the specified regions. +- **Node Status** — The number of nodes up and running as compared to the available nodes in the specified region in a text chart. It provides the status of all nodes in the specified region using boolean indicator (green (OK)/red (KO)) in a text chart. + +On expanding each item, it provides a list nodes with information like: +- Total number of active connections as compared to the maximum number of configured connections for each node. +- Memory usage percentage on a progress bar. +- Storage usage percentage on a progress bar. + +For PGD, it provides the tags below the node name: +- **Raft Leader**— It indicates that the node is a Raft Leader locally in the region. +- **Raft Follower**— It indicates that the node is a Raft Follower locally in the region. +- **Global Raft Leader**— It indicates that the nodes is a Raft Leader globally in the cluster. +- **Global Raft Follower**— It indicates that the node is a Raft Follower globally in the cluster. +- **Witness**— It indicates that the node is a witness in the PGD cluster. See [witness node docs](/pgd/latest/node_management/witness_nodes/) for more information. + +For high-availability, it provides the tags below the node name: +- **Primary**— It indicates if the node role is primary. + +### Replication status + +The Replication Status section has a matrix displaying the replication lag across all the nodes of the cluster. The matrix provides different types of replication lags for **Write**, **Replay**, **Flush**, and **Sent**. It provides the lag in both bytes and time for **Write**, **Replay**, and **Flush** whereas only in bytes for **Sent**. + +!!!note +- In high-availability clusters, the replication occurs only from the primary(source) to replicas(target). So the matrix displays only one row for the source and multiple columns for the targets. +- In PGD clusters, the replications occurs bidirectional. So the matrix displays equal number of rows and columns (every node is both source and target). +!!! + +!!!note +The data on the Health Status dashboard is dynamic and keeps updated continuously. The structure of the accordion is based on a snapshot of the cluster architecture. If any new node is added or existing node is removed, then reload the Health Status dashboard manually by hitting the tab again or reload the browser page. +!!! + From e645f4f965f12fa617efc6522ff7a6e0f27463fe Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 21 May 2024 13:57:46 +0100 Subject: [PATCH 07/67] Updated with new error Signed-off-by: Dj Walker-Morgan --- .../migration_toolkit/55/09_mtk_errors.mdx | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx b/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx index 4a55ae39641..823be8d3c89 100644 --- a/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx +++ b/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx @@ -256,3 +256,22 @@ at java.lang.Thread.run(Thread.java:722) By default, the MySQL JDBC driver fetches all of the rows in a table into Migration Toolkit in a single network round trip. This behavior can easily exceed available memory for large tables. To correct the problem, specify `-fetchSize 1` as a command-line argument when you retry the migration. + +#### Incorrect timestamps and Daylight Saving Time + +When migrating from SQL Server to PostgreSQL using the MSSQL JDBC driver there may be no errors observed. If the source database contains rows with timestamp values which are within a range of time when Daylight Savings Time was in effect, they will be migrated to the target database with the wrong timestamps. + +To resolve this issue you can update the `runMTK.sh` file and provide the option `-Duser.timezone=GMT`. This will be in operation then when running the toolkit, + +For example, given the original line: + +```text +runJREApplication $JAVA_HEAP_SIZE -Dprop=$base/etc/toolkit.properties -cp $base/bin/edb-migrationtoolkit.jar:$base/lib/* com.edb.MigrationToolkit "$@" +``` + +This should be updated with the `-Duser.timezone=GMT` inserted before the `-cp` option: + +```text +runJREApplication $JAVA_HEAP_SIZE -Dprop=$base/etc/toolkit.properties -Duser.timezone=GMT -cp $base/bin/edb-migrationtoolkit.jar:$base/lib/* com.edb.MigrationToolkit "$@" +``` + From aad3a9ca2b2757bce305aac50a9cbf030494a788 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 21 May 2024 17:38:20 +0100 Subject: [PATCH 08/67] Added deeptoc and balanced headings Signed-off-by: Dj Walker-Morgan --- product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx b/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx index 823be8d3c89..3d31145e9a3 100644 --- a/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx +++ b/product_docs/docs/migration_toolkit/55/09_mtk_errors.mdx @@ -1,6 +1,6 @@ --- title: "Migration errors" - +deepToC: true --- @@ -204,7 +204,7 @@ Schema HR imported with errors. The table definition in the target database doesn't match the migrated data. If you altered the target or source table, confirm that the table definition and the data are compatible. -#### ERROR: value too long for type +### ERROR: value too long for type *I've already migrated the table definition. When I try to migrate the data into the table, I get the following error:* @@ -230,7 +230,7 @@ Where: `COPY departments, line 1, column location_id: "1700"` To correct the problem, adjust the column size and retry the migration. -#### ERROR: Exception in thread:OutOfMemoryError +### ERROR: Exception in thread:OutOfMemoryError *When migrating from a MySQL database, I encounter the following error:* @@ -257,7 +257,7 @@ By default, the MySQL JDBC driver fetches all of the rows in a table into Migrat To correct the problem, specify `-fetchSize 1` as a command-line argument when you retry the migration. -#### Incorrect timestamps and Daylight Saving Time +### Incorrect timestamps and Daylight Saving Time When migrating from SQL Server to PostgreSQL using the MSSQL JDBC driver there may be no errors observed. If the source database contains rows with timestamp values which are within a range of time when Daylight Savings Time was in effect, they will be migrated to the target database with the wrong timestamps. From 9f5c0a85e3fe39b4ceab274e14965f3136604941 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Wed, 22 May 2024 17:41:45 +0530 Subject: [PATCH 09/67] minor edit --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 0ea28a39b9a..1fdcba827fb 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -14,7 +14,7 @@ Health Status dashboard displays: To view the **Health Status** dashboard from the BigAnimal portal: 1. In the left navigation of BigAnimal Portal, go to **Clusters**. -2. Select any ready cluster. +2. Select any ready high-availability or PGD cluster. 3. On the cluster detail page, select the **Health Status** tab. The **Health Status** tab displays the dashboard with health status categorized in sections: From 6cfc443c363a2525a25677e4c2357503bdd81834 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Wed, 22 May 2024 12:20:02 +0100 Subject: [PATCH 10/67] Update upgrade_paths.mdx --- product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx index fda5504a23a..b79527b6947 100644 --- a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx +++ b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx @@ -37,7 +37,8 @@ following upgrade paths are possible. | 3.7.19 | 4.2.2 only | | 3.7.20 | 4.3.0 to 4.3.2 | | 3.7.21 | 4.3.1 to 4.3.2 | -| 3.7.22 | 4.3.2 only | +| 3.7.22 | 4.3.2 or later | +| 3.7.23 | 4.3.3 or later | ## Upgrading from version 3.6 to version 4 From 3ec546a0fb5699b21f745325f5c555759887cae5 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:07:43 +0530 Subject: [PATCH 11/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 1fdcba827fb..69fbf722da3 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -61,7 +61,7 @@ For high-availability, it provides the tags below the node name: The Replication Status section has a matrix displaying the replication lag across all the nodes of the cluster. The matrix provides different types of replication lags for **Write**, **Replay**, **Flush**, and **Sent**. It provides the lag in both bytes and time for **Write**, **Replay**, and **Flush** whereas only in bytes for **Sent**. !!!note -- In high-availability clusters, the replication occurs only from the primary(source) to replicas(target). So the matrix displays only one row for the source and multiple columns for the targets. +- In high-availability clusters, replication occurs only from the primary (source) to the replicas (target). So the matrix displays only one row for the source and multiple columns for the targets. - In PGD clusters, the replications occurs bidirectional. So the matrix displays equal number of rows and columns (every node is both source and target). !!! From 4c90deff9c73e456569d5e7c33bc654ee00fccfd Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:08:41 +0530 Subject: [PATCH 12/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 69fbf722da3..865a9c7e3c2 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -62,7 +62,7 @@ The Replication Status section has a matrix displaying the replication lag acros !!!note - In high-availability clusters, replication occurs only from the primary (source) to the replicas (target). So the matrix displays only one row for the source and multiple columns for the targets. -- In PGD clusters, the replications occurs bidirectional. So the matrix displays equal number of rows and columns (every node is both source and target). +- In PGD clusters, the replications are bidirectional, so the matrix displays an equal number of rows and columns. Every node is both a source and a target. !!! !!!note From 8c6ef28a45587e64e1453b6b686d147374bf5728 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:11:16 +0530 Subject: [PATCH 13/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 865a9c7e3c2..afcaa8bb752 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -66,6 +66,6 @@ The Replication Status section has a matrix displaying the replication lag acros !!! !!!note -The data on the Health Status dashboard is dynamic and keeps updated continuously. The structure of the accordion is based on a snapshot of the cluster architecture. If any new node is added or existing node is removed, then reload the Health Status dashboard manually by hitting the tab again or reload the browser page. +The data on the Health Status dashboard is dynamic and is updated continuously. However, the cluster architecture is based on a snapshot. If a new node is added or an existing node is removed, you must reload the Health Status dashboard by clicking the tab again or reloading the browser page. !!! From 9c9bc81bc846d0ebd039f24dce874cad350bd71d Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:12:21 +0530 Subject: [PATCH 14/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index afcaa8bb752..52d6a716157 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -2,7 +2,7 @@ title: "Health Status" --- -Health Status dashboard provides the status of the Postgres high-availability clusters. It provides status for both Primary/Standby and Distributed high-availability clusters. +The Health Status dashboard provides real-time insight into the topology and health of Postgres high-availability clusters. It supports both Primary/Standby and Distributed high-availability clusters. Health Status dashboard displays: - A set of cluster-wide health indicators that helps to draw your attention immediately to the critical issues. From 9331db1561e251d9fd8375c8ea936e4461ba3954 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:12:32 +0530 Subject: [PATCH 15/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 52d6a716157..d32d6f400a2 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -4,7 +4,7 @@ title: "Health Status" The Health Status dashboard provides real-time insight into the topology and health of Postgres high-availability clusters. It supports both Primary/Standby and Distributed high-availability clusters. -Health Status dashboard displays: +The Health Status dashboard displays: - A set of cluster-wide health indicators that helps to draw your attention immediately to the critical issues. - A schematic view of all the nodes organized in a cluster distributed across regions displaying health and role(s) of each node. - A replication status view displaying the status of each replication slot and the associated replication lags. From 3abdae0ef30d14f8fa072fd1d1f3ee0213971732 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:13:01 +0530 Subject: [PATCH 16/67] Update product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index d32d6f400a2..12b13fc624a 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -37,7 +37,7 @@ The global cluster health section displays the cluster-wide view including the f ### Regional nodes health and roles -The regional nodes health and roles section displays fine-grained health status at regional and node level. It is structured as an accordion, with each element representing a group of nodes deployed in the same region. Each item displays basic information like: +The regional nodes health and roles section displays fine-grained health status at regional and node level. It is structured as an accordion, with each element representing a group of nodes deployed in the same region. Each item displays basic information including: - **Proxy Status** (only PGD)— It indicates the number of active proxies compared to the available proxies in the specified regions. - **Node Status** — The number of nodes up and running as compared to the available nodes in the specified region in a text chart. It provides the status of all nodes in the specified region using boolean indicator (green (OK)/red (KO)) in a text chart. From 8423cee6867fe89ce313a2ca81c1c48307bdaf38 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:17:43 +0530 Subject: [PATCH 17/67] updated content as per suggestion from Simon --- .../health_status.mdx | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 12b13fc624a..97d58f99e2f 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -27,21 +27,21 @@ The **Health Status** tab displays the dashboard with health status categorized The global cluster health section displays the cluster-wide view including the following metrics: -- **Raft Status** (PGD only)— It indicates whether the Raft consensus algorithm is running correctly in the cluster. It verifies that one node is elected as the global leader and the Raft roles such as RAFT_LEADER and RAFT_FOLLOWER are defined correctly across all the nodes. -- **Replication Slot Status** (PGD only)— It indicates whether all the replication slots are in streaming state or not. -- **Clock Skew**— It indicates whether the node's clock is in sync and doesn't exceed a threshold of 60 seconds. -- **Proxy Status** (PGD only)— It provides the number of PGD proxies up and running as compared to the available proxies. -- **Node Status**— It provides the number of nodes that are up and running. -- **Transaction rate**— It provides the total number of transactions including committed and rolled back transactions per second in the cluster. +- **Raft Status** (PGD only) indicates whether the Raft consensus algorithm is running correctly in the cluster. It verifies that one node is elected as the global leader and the Raft roles such as RAFT_LEADER and RAFT_FOLLOWER are defined correctly across all the nodes. +- **Replication Slot Status** (PGD only) indicates whether all the replication slots are in streaming state or not. +- **Clock Skew** indicates whether the node's clock is in sync and doesn't exceed a threshold of 60 seconds. +- **Proxy Status** (PGD only) provides the number of PGD proxies up and running as compared to the available proxies. +- **Node Status** provides the number of nodes that are up and running. +- **Transaction rate** provides the total number of transactions including committed and rolled back transactions per second in the cluster. ### Regional nodes health and roles The regional nodes health and roles section displays fine-grained health status at regional and node level. It is structured as an accordion, with each element representing a group of nodes deployed in the same region. Each item displays basic information including: -- **Proxy Status** (only PGD)— It indicates the number of active proxies compared to the available proxies in the specified regions. -- **Node Status** — The number of nodes up and running as compared to the available nodes in the specified region in a text chart. It provides the status of all nodes in the specified region using boolean indicator (green (OK)/red (KO)) in a text chart. +- **Proxy Status** (only PGD) indicates the number of active proxies compared to the available proxies in the specified regions. +- **Node Status** indicates the number of nodes up and running as compared to the available nodes in the specified region in a text chart. It provides the status of all nodes in the specified region using boolean indicator (green (OK)/red (KO)) in a text chart. -On expanding each item, it provides a list nodes with information like: +On expanding each item, it provides a list of nodes with information like: - Total number of active connections as compared to the maximum number of configured connections for each node. - Memory usage percentage on a progress bar. - Storage usage percentage on a progress bar. From 9ee753bec36f86fa5a0f3d1adad69841a23e2a8b Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:19:43 +0530 Subject: [PATCH 18/67] added deepToc --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 97d58f99e2f..7616a9139ed 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -1,5 +1,6 @@ --- title: "Health Status" +deepToC: true --- The Health Status dashboard provides real-time insight into the topology and health of Postgres high-availability clusters. It supports both Primary/Standby and Distributed high-availability clusters. From 47c9ff44acef2cf19aa723b70b43527bd1c90c67 Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 09:26:34 +0530 Subject: [PATCH 19/67] clean up --- .../health_status.mdx | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index 7616a9139ed..c8a80cf0b07 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -15,14 +15,15 @@ The Health Status dashboard displays: To view the **Health Status** dashboard from the BigAnimal portal: 1. In the left navigation of BigAnimal Portal, go to **Clusters**. + 2. Select any ready high-availability or PGD cluster. + 3. On the cluster detail page, select the **Health Status** tab. The **Health Status** tab displays the dashboard with health status categorized in sections: -- [Viewing Health Status dashboard](#viewing-health-status-dashboard) - - [Global cluster health](#global-cluster-health) - - [Regional nodes health and roles](#regional-nodes-health-and-roles) - - [Replication status](#replication-status) +- [Global cluster health](#global-cluster-health) +- [Regional nodes health and roles](#regional-nodes-health-and-roles) +- [Replication status](#replication-status) ### Global cluster health @@ -48,14 +49,14 @@ On expanding each item, it provides a list of nodes with information like: - Storage usage percentage on a progress bar. For PGD, it provides the tags below the node name: -- **Raft Leader**— It indicates that the node is a Raft Leader locally in the region. -- **Raft Follower**— It indicates that the node is a Raft Follower locally in the region. -- **Global Raft Leader**— It indicates that the nodes is a Raft Leader globally in the cluster. -- **Global Raft Follower**— It indicates that the node is a Raft Follower globally in the cluster. -- **Witness**— It indicates that the node is a witness in the PGD cluster. See [witness node docs](/pgd/latest/node_management/witness_nodes/) for more information. +- **Raft Leader** indicates that the node is a Raft Leader locally in the region. +- **Raft Follower** indicates that the node is a Raft Follower locally in the region. +- **Global Raft Leader** indicates that the nodes is a Raft Leader globally in the cluster. +- **Global Raft Follower** indicates that the node is a Raft Follower globally in the cluster. +- **Witness** indicates that the node is a witness in the PGD cluster. See [witness node docs](/pgd/latest/node_management/witness_nodes/) for more information. For high-availability, it provides the tags below the node name: -- **Primary**— It indicates if the node role is primary. +- **Primary** indicates if the node role is primary. ### Replication status From 15ac017725bfe2c8e10ee5590b70fd37f67cd15b Mon Sep 17 00:00:00 2001 From: nidhibhammar <59045594+nidhibhammar@users.noreply.github.com> Date: Thu, 23 May 2024 13:40:57 +0530 Subject: [PATCH 20/67] Added "Node Ko" as per @aviola's suggestion --- .../using_cluster/05_monitoring_and_logging/health_status.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx index c8a80cf0b07..f3e78c1028d 100644 --- a/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx +++ b/product_docs/docs/biganimal/release/using_cluster/05_monitoring_and_logging/health_status.mdx @@ -45,6 +45,7 @@ The regional nodes health and roles section displays fine-grained health status On expanding each item, it provides a list of nodes with information like: - Total number of active connections as compared to the maximum number of configured connections for each node. +- A **Node Ko** tag for each node if it is down. - Memory usage percentage on a progress bar. - Storage usage percentage on a progress bar. From d21a86930f51ec3819cdc3a921719fe1cda391af Mon Sep 17 00:00:00 2001 From: Josh Heyer Date: Wed, 22 May 2024 15:10:10 -0700 Subject: [PATCH 21/67] Structure for new docs --- advocacy_docs/postgres-ai/ai-ml/index.mdx | 5 +++++ advocacy_docs/postgres-ai/analytics/index.mdx | 5 +++++ advocacy_docs/postgres-ai/console/index.mdx | 5 +++++ advocacy_docs/postgres-ai/databases/index.mdx | 5 +++++ advocacy_docs/postgres-ai/index.mdx | 12 ++++++++++++ advocacy_docs/postgres-ai/tools/index.mdx | 5 +++++ src/pages/index.js | 11 +++++++++++ 7 files changed, 48 insertions(+) create mode 100644 advocacy_docs/postgres-ai/ai-ml/index.mdx create mode 100644 advocacy_docs/postgres-ai/analytics/index.mdx create mode 100644 advocacy_docs/postgres-ai/console/index.mdx create mode 100644 advocacy_docs/postgres-ai/databases/index.mdx create mode 100644 advocacy_docs/postgres-ai/index.mdx create mode 100644 advocacy_docs/postgres-ai/tools/index.mdx diff --git a/advocacy_docs/postgres-ai/ai-ml/index.mdx b/advocacy_docs/postgres-ai/ai-ml/index.mdx new file mode 100644 index 00000000000..073269a40e5 --- /dev/null +++ b/advocacy_docs/postgres-ai/ai-ml/index.mdx @@ -0,0 +1,5 @@ +--- +title: Postgres AI - AI-ML +navTitle: AI-ML +iconName: Learning +--- diff --git a/advocacy_docs/postgres-ai/analytics/index.mdx b/advocacy_docs/postgres-ai/analytics/index.mdx new file mode 100644 index 00000000000..c20a3b53944 --- /dev/null +++ b/advocacy_docs/postgres-ai/analytics/index.mdx @@ -0,0 +1,5 @@ +--- +title: Postgres AI - Analytics +navTitle: Analytics +iconName: CrystalBall +--- diff --git a/advocacy_docs/postgres-ai/console/index.mdx b/advocacy_docs/postgres-ai/console/index.mdx new file mode 100644 index 00000000000..af37ac0f826 --- /dev/null +++ b/advocacy_docs/postgres-ai/console/index.mdx @@ -0,0 +1,5 @@ +--- +title: Postgres AI - Console +navTitle: Console +iconName: Control +--- diff --git a/advocacy_docs/postgres-ai/databases/index.mdx b/advocacy_docs/postgres-ai/databases/index.mdx new file mode 100644 index 00000000000..154e2265087 --- /dev/null +++ b/advocacy_docs/postgres-ai/databases/index.mdx @@ -0,0 +1,5 @@ +--- +title: Postgres AI - Databases +navTitle: Databases +iconName: Database +--- diff --git a/advocacy_docs/postgres-ai/index.mdx b/advocacy_docs/postgres-ai/index.mdx new file mode 100644 index 00000000000..7f5d29717bc --- /dev/null +++ b/advocacy_docs/postgres-ai/index.mdx @@ -0,0 +1,12 @@ +--- +title: Postgres AI +directoryDefaults: + iconName: BrainCircuit + indexCards: full +navigation: + - console + - databases + - analytics + - ai-ml + - tools +--- diff --git a/advocacy_docs/postgres-ai/tools/index.mdx b/advocacy_docs/postgres-ai/tools/index.mdx new file mode 100644 index 00000000000..8ddd8ca4d65 --- /dev/null +++ b/advocacy_docs/postgres-ai/tools/index.mdx @@ -0,0 +1,5 @@ +--- +title: Postgres AI - Tools +navTitle: Tools +iconName: Toolbox +--- diff --git a/src/pages/index.js b/src/pages/index.js index 5f56788aa16..2199cedaa18 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -97,6 +97,17 @@ const Page = () => { + {/* Postgres AI nav goes here */} + +
+ Postgres AI nav goes here + Console + Databases + Analytics + AI-ML + Tools +
+ {/* Sign Post */}
From e75ec76e4d01a5042ffb8c164e028e306b84daee Mon Sep 17 00:00:00 2001 From: Josh Heyer Date: Wed, 15 May 2024 10:26:03 -0700 Subject: [PATCH 22/67] Per Catherine, edb prefix mandatory --- .../ai-ml/index.mdx | 0 .../analytics/index.mdx | 0 .../console/index.mdx | 0 .../databases/index.mdx | 0 .../{postgres-ai => edb-postgres-ai}/index.mdx | 0 .../tools/index.mdx | 0 src/pages/index.js | 16 ++++++++++------ 7 files changed, 10 insertions(+), 6 deletions(-) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/ai-ml/index.mdx (100%) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/analytics/index.mdx (100%) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/console/index.mdx (100%) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/databases/index.mdx (100%) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/index.mdx (100%) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/tools/index.mdx (100%) diff --git a/advocacy_docs/postgres-ai/ai-ml/index.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/ai-ml/index.mdx rename to advocacy_docs/edb-postgres-ai/ai-ml/index.mdx diff --git a/advocacy_docs/postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/analytics/index.mdx rename to advocacy_docs/edb-postgres-ai/analytics/index.mdx diff --git a/advocacy_docs/postgres-ai/console/index.mdx b/advocacy_docs/edb-postgres-ai/console/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/console/index.mdx rename to advocacy_docs/edb-postgres-ai/console/index.mdx diff --git a/advocacy_docs/postgres-ai/databases/index.mdx b/advocacy_docs/edb-postgres-ai/databases/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/databases/index.mdx rename to advocacy_docs/edb-postgres-ai/databases/index.mdx diff --git a/advocacy_docs/postgres-ai/index.mdx b/advocacy_docs/edb-postgres-ai/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/index.mdx rename to advocacy_docs/edb-postgres-ai/index.mdx diff --git a/advocacy_docs/postgres-ai/tools/index.mdx b/advocacy_docs/edb-postgres-ai/tools/index.mdx similarity index 100% rename from advocacy_docs/postgres-ai/tools/index.mdx rename to advocacy_docs/edb-postgres-ai/tools/index.mdx diff --git a/src/pages/index.js b/src/pages/index.js index 2199cedaa18..8df73d76fdc 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -100,12 +100,16 @@ const Page = () => { {/* Postgres AI nav goes here */}
- Postgres AI nav goes here - Console - Databases - Analytics - AI-ML - Tools +

Postgres AI nav goes here

+ Console + + Databases + + Analytics + + AI-ML + + Tools
{/* Sign Post */} From 60bee78ccefc140dda6047d06d156fc4799f2e60 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Sat, 18 May 2024 15:19:39 +0100 Subject: [PATCH 23/67] Dj Draft Done --- .../ai-ml/databases-and-ai.mdx | 7 + advocacy_docs/edb_postgres_ai/ai-ml/index.mdx | 13 ++ .../edb_postgres_ai/ai-ml/overview.mdx | 11 ++ .../edb_postgres_ai/ai-ml/postgres-and-ai.mdx | 7 + .../edb_postgres_ai/analytics/analytics.mdx | 9 ++ .../edb_postgres_ai/analytics/getstarted.mdx | 10 ++ .../edb_postgres_ai/analytics/index.mdx | 14 ++ .../edb_postgres_ai/analytics/lakehouse.mdx | 9 ++ .../edb_postgres_ai/console/agent.mdx | 10 ++ .../edb_postgres_ai/console/estate.mdx | 9 ++ .../edb_postgres_ai/console/getstarted.mdx | 9 ++ .../edb_postgres_ai/console/index.mdx | 13 ++ .../databases/cloudservice.mdx | 9 ++ .../edb_postgres_ai/databases/index.mdx | 12 ++ .../edb_postgres_ai/databases/onprem.mdx | 10 ++ .../edb_postgres_ai/databases/options.mdx | 10 ++ advocacy_docs/edb_postgres_ai/index.mdx | 15 ++ .../edb_postgres_ai/overview/concepts.mdx | 53 +++++++ .../edb_postgres_ai/overview/features.mdx | 11 ++ .../edb_postgres_ai/overview/guide.mdx | 6 + .../edb_postgres_ai/overview/index.mdx | 10 ++ .../edb_postgres_ai/tools/backup.mdx | 10 ++ advocacy_docs/edb_postgres_ai/tools/index.mdx | 16 ++ .../edb_postgres_ai/tools/management.mdx | 11 ++ .../tools/migration-and-ai.mdx | 13 ++ build-sources.json | 3 +- gatsby-config.js | 4 + src/components/icon/iconNames.js | 3 + src/components/icon/index.js | 3 + src/components/left-nav.js | 11 +- src/components/logo.js | 2 +- src/constants/products.js | 4 + src/pages/index.js | 144 ++++++++++++++++++ static/icons/edb_landscape_color_grey.svg | 44 ++++++ 34 files changed, 521 insertions(+), 4 deletions(-) create mode 100644 advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx create mode 100644 advocacy_docs/edb_postgres_ai/ai-ml/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx create mode 100644 advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx create mode 100644 advocacy_docs/edb_postgres_ai/analytics/analytics.mdx create mode 100644 advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx create mode 100644 advocacy_docs/edb_postgres_ai/analytics/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx create mode 100644 advocacy_docs/edb_postgres_ai/console/agent.mdx create mode 100644 advocacy_docs/edb_postgres_ai/console/estate.mdx create mode 100644 advocacy_docs/edb_postgres_ai/console/getstarted.mdx create mode 100644 advocacy_docs/edb_postgres_ai/console/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx create mode 100644 advocacy_docs/edb_postgres_ai/databases/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/databases/onprem.mdx create mode 100644 advocacy_docs/edb_postgres_ai/databases/options.mdx create mode 100644 advocacy_docs/edb_postgres_ai/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/overview/concepts.mdx create mode 100644 advocacy_docs/edb_postgres_ai/overview/features.mdx create mode 100644 advocacy_docs/edb_postgres_ai/overview/guide.mdx create mode 100644 advocacy_docs/edb_postgres_ai/overview/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/tools/backup.mdx create mode 100644 advocacy_docs/edb_postgres_ai/tools/index.mdx create mode 100644 advocacy_docs/edb_postgres_ai/tools/management.mdx create mode 100644 advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx create mode 100644 static/icons/edb_landscape_color_grey.svg diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx b/advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx new file mode 100644 index 00000000000..9cb0c30858e --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx @@ -0,0 +1,7 @@ +--- +title: EDB Postgres AI AI/ML - Databases and AI +navTitle: Databases and AI +description: An introduction to the intersection of databases and AI/ML with EDB Postgres AI. +--- + +_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!__ diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx b/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx new file mode 100644 index 00000000000..fe53c8645bf --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx @@ -0,0 +1,13 @@ +--- +title: EDB Postgres AI - AI/ML +navTitle: AI/ML +indexCards: simple +navigation: +- overview +- databases-and-ai +- postgres-and-ai +--- + +EDB Postgres AI AI/ML. + +_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx b/advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx new file mode 100644 index 00000000000..6149bda5fa4 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx @@ -0,0 +1,11 @@ +--- +title: EDB Postgres AI AI/ML - Overview +navTitle: Overview +description: Where to start with EDB Postgres AI AI/ML, from creating innovative models to putting them into production. +--- + +Where to start with EDB Postgres AI AI/ML, from creating innovative models to putting them into production. + +_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ + + diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx b/advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx new file mode 100644 index 00000000000..1ca96dc4512 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx @@ -0,0 +1,7 @@ +--- +title: EDB Postgres AI AI/ML - Postgres and AI +navTitle: Postgres and AI +description: From the general to the specific, this section provides an overview of the intersection of Postgres and AI/ML with EDB Postgres AI. +--- + +_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!_ diff --git a/advocacy_docs/edb_postgres_ai/analytics/analytics.mdx b/advocacy_docs/edb_postgres_ai/analytics/analytics.mdx new file mode 100644 index 00000000000..9e10591e39b --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/analytics/analytics.mdx @@ -0,0 +1,9 @@ +--- +title: EDB Postgres AI Analytics - Analytics +navTitle: Analytics +description: Discover how EDB Postgres AI Analytics can help you turn data into insights. +--- + +An Introduction to EDB Postgres AI Analytics. + +_Corrall your data and then deep dive into it with EDB Postgres AI Analytics. This section provides an overview of the tools and features that help you to gather, analyze, and visualize data. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ diff --git a/advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx b/advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx new file mode 100644 index 00000000000..9fc3a50c48c --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Analytics - Get Started +navTitle: Get Started +description: Where to begin with EDB Postgres AI Analytics and how to get results quickly. +--- + +Where to begin with EDB Postgres AI Analytics and how to get results quickly. + +_The EDB Postgres AI Analytics is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for analytics workloads. The service is built on the EDB Postgres Advanced Server database and is designed to help organizations accelerate the development and deployment of analytics applications._ + diff --git a/advocacy_docs/edb_postgres_ai/analytics/index.mdx b/advocacy_docs/edb_postgres_ai/analytics/index.mdx new file mode 100644 index 00000000000..b286cad418c --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/analytics/index.mdx @@ -0,0 +1,14 @@ +--- +title: EDB Postgres AI Analytics +navTitle: Analytics +indexCards: simple +navigation: +- getstarted +- analytics +- lakehouse +--- + +Introduction to EDB Postgres AI Analytics. + +_EDB Postgres AI Analytics provides the tools needed to gather, analyze, and visualize data. It includes a variety of tools and features that help you to analyze data and make data-driven decisions. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ + diff --git a/advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx b/advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx new file mode 100644 index 00000000000..ed27d019715 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx @@ -0,0 +1,9 @@ +--- +title: EDB Postgres AI Databases- Lakehouse +navTitle: Lakehouse +description: The EDB Postgres AI Lakehouse provides a single source of truth for all your data. +--- + +_Bring all your data together with the EDB Postgres AI Lakehouse. Get a single source of truth for your Analytics needs. The Lakehouse is a unified data platform that combines the best of data lakes and data warehouses. With a single source of truth for all your data, you are able to analyze and visualize data in real time_ + + diff --git a/advocacy_docs/edb_postgres_ai/console/agent.mdx b/advocacy_docs/edb_postgres_ai/console/agent.mdx new file mode 100644 index 00000000000..3a964e51fe7 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/console/agent.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Console - Agent +navTitle: Agent +description: The EDB Postgres AI Console Agent allows you to expand your EDB Postgres AI Console's visibility to your on-premises and in-cloud Postgres databases. +--- + +## What is the EDB Postgres AI Agent + +_The EDB Postgres AI Agent is a component of the EDB Postgres AI Console. The Agent is installed alongside your existing On-prem or in-cloud Postgres databases and makes them observable on the EDB Postgres AI Console's single pane of glass view._ + diff --git a/advocacy_docs/edb_postgres_ai/console/estate.mdx b/advocacy_docs/edb_postgres_ai/console/estate.mdx new file mode 100644 index 00000000000..7e53bcd2366 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/console/estate.mdx @@ -0,0 +1,9 @@ +--- +title: EDB Postgres AI Console - Estate +navTitle: Estate +description: How to manage and integrate EDB Postgres AI Databases and more with EDB Postgres AI Console's single pane of glass. +--- + +## What is EDB Postgres AI Estate + +_The EDB Postgres AI Estate is a component of the EDB Postgres AI Console that provides a single pane of glass for managing and integrating EDB Postgres AI Databases and EDB Postgres AI Agents. The Estate provides a centralized location for managing the lifecycle of EDB Postgres AI Databases and EDB Postgres AI Agents, including provisioning, scaling, and monitoring. The Estate also provides a centralized location for managing the integration of EDB Postgres AI Databases and EDB Postgres AI Agents with the EDB Postgres AI Console's single pane of glass._ \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/console/getstarted.mdx b/advocacy_docs/edb_postgres_ai/console/getstarted.mdx new file mode 100644 index 00000000000..4c4a2e5c1d1 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/console/getstarted.mdx @@ -0,0 +1,9 @@ +--- +title: EDB Postgres AI Console - Get Started +navTitle: Get Started +description: Get started with the EDB Postgres AI Console. +--- + +Getting Started with the EDB Postgres AI Console. + +_The EDB Postgres AI Console is a web-based user interface that provides a single pane of glass for managing and monitoring EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases. The EDB Postgres AI Console provides a unified view of the EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases, allowing users to manage and monitor their databases, users, and resources from a single interface._ \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/console/index.mdx b/advocacy_docs/edb_postgres_ai/console/index.mdx new file mode 100644 index 00000000000..84e05c0c506 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/console/index.mdx @@ -0,0 +1,13 @@ +--- +title: EDB Postgres AI Console +navTitle: Console +indexCards: simple +navigation: +- getstarted +- estate +- agent +--- + +Introduction to the EDB Postgres AI Console + +_The EDB Postgres AI Console is a web-based interface that allows you to manage your EDB Postgres AI environment. The console provides a user-friendly interface for managing your estate, including the remote agents and databases that run on your servers._ diff --git a/advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx b/advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx new file mode 100644 index 00000000000..3936387d141 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx @@ -0,0 +1,9 @@ +--- +title: EDB Postgres AI Cloud Service +navTitle: Cloud Service +description: An introduction to the EDB Postgres AI Cloud Service and its features. +--- + +Introduction to the EDB Postgres AI Cloud Service + +_The EDB Postgres AI Cloud Service is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for AI and machine learning workloads. The service is built on the EDB Postgres Advanced Server database and is designed to help organizations accelerate the development and deployment of AI and machine learning applications._ diff --git a/advocacy_docs/edb_postgres_ai/databases/index.mdx b/advocacy_docs/edb_postgres_ai/databases/index.mdx new file mode 100644 index 00000000000..f9878718643 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/databases/index.mdx @@ -0,0 +1,12 @@ +--- +title: EDB Postgres AI Databases +navTitle: Databases +indexCards: simple +--- + +Introduction to the EDB Postgres AI Databases and the EDB Postgres AI Cloud Service + +_The EDB Postgres AI Databases are a set of databases that are optimized for machine learning and artificial intelligence workloads. The databases are designed to provide high performance and scalability for machine learning and artificial intelligence workloads. The EDB Postgres AI Databases are available as part of the EDB Postgres AI Cloud Service._ + + + diff --git a/advocacy_docs/edb_postgres_ai/databases/onprem.mdx b/advocacy_docs/edb_postgres_ai/databases/onprem.mdx new file mode 100644 index 00000000000..187c4977150 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/databases/onprem.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Databases- On Premises +navTitle: On Premises +description: Deploy EDB Postgres AI Databases on-premises with the EDB Postgres AI Estate and Agent components. +--- + +EDB Postgres AI Databases are available for on-premises deployment. + +_The on-premises deployment of EDB Postgres AI Database are a self-managed deployment, which can use the EDB Postgres AI Estate and Agent components to manage and integrate the on-premises deployment alongside the EDB Postgres AI Database Cloud Service through the EDB Postgres AI Console's single pane of glass._ + diff --git a/advocacy_docs/edb_postgres_ai/databases/options.mdx b/advocacy_docs/edb_postgres_ai/databases/options.mdx new file mode 100644 index 00000000000..e51101919a7 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/databases/options.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Databases - Optional Features +navTitle: Optional Features +description: High Availability and other available for EDB Postgres AI Databases and on EDB Postgres AI Cloud Service. +--- + +Introducing EDB Postgres Distributed and other optional features available for EDB Postgres AI Databases. + +_EDB Postgres Distributed is a separate product that is available for EDB Postgres AI Databases. It provides a distributed SQL database that is designed to support high availability. EDB Postgres Distributed powers the EDB Postgres AI Cloud Service's distributed high availability option._ + diff --git a/advocacy_docs/edb_postgres_ai/index.mdx b/advocacy_docs/edb_postgres_ai/index.mdx new file mode 100644 index 00000000000..4b9aaaa93ee --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/index.mdx @@ -0,0 +1,15 @@ +--- +title: EDB Postgres AI Home +navTitle: Home +indexCards: full +navigation: +- overview +- console +- databases +- analytics +- ai-ml +- tools +--- + +The future home of all EDB Postgres AI documentation. Stay tuned! + diff --git a/advocacy_docs/edb_postgres_ai/overview/concepts.mdx b/advocacy_docs/edb_postgres_ai/overview/concepts.mdx new file mode 100644 index 00000000000..e434bf4dc3d --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/overview/concepts.mdx @@ -0,0 +1,53 @@ +--- +title: EDB Postgres AI Overview - Concepts +navTitle: Concepts +deepToc: true +description: A look at the concepts that underpin EDB Postgres AI. +--- + +## What is EDB Postgres AI? + +EDB Postgres AI takes EDB’s leading expertise in Postgres and expands the scope of Postgres to address modern challenges. From simplifying your database estate management to infusing AI deep into Postgres and putting it to work to bring all your data under one analytical eye. + +EDB Postgres AI is composed of multiple elements which come together to deliver a unified and powerful experience: + +### EDB Postgres AI - Console: + +* Providing the single pane of glass onto all your EDB Postgres AI operations, Console is there to manage your database landscape. + * #### Integrated management: + * EDB Postgres AI Cloud databases are automatically managed on the Console. + * #### EDB Postgres AI Agent: + * On premises databases can be brought under one manageable Console view with the Agent enabling an unprecedented view of diverse deployments. + +### EDB Postgres AI - Databases: + +* All of EDB’s database expertise can be found in EDB Postgres Advanced Server and EDB Postgres Extended Server. +* Oracle compatibility, transparent data encryption and more. They provide the data fabric on which EDB Postgres AI operates. +* Combined with EDB Postgres Distributed, they can also provide a high availability environment for your data. +* All of these components are available on the EDB Postgres AI Cloud Service, and managed through the EDB Postgres AI Console. + * #### EDB Postgres Advanced Server and EDB Postgres Extended Server: + * … + * #### EDB Postgres Distributed: + * … + * #### EDB Postgres AI Cloud Service: + * Not just databases, but driven by databases, Cloud Service provides a global platform for delivering new elements of EDB Postgres AI efficiently and effectively. + +### EDB Postgres AI - Analytics: + +* Filtering out the data noise and revealing insights and value, Analytics brings both structured relational data in Postgres and unstructured data in object storage together for exploration. At the heart of Analytics is a custom built store for this data: + +* ### EDB Postgres AI - Lakehouse: + * Built to bring structured and unstructured data together, Lakehouse supports numerous formats to bring your data in from the cold, ready to be analyzed. + +### EDB Postgres AI - AI/ML: + +* Postgres has proven its capability as a flexible data environment, and Vector data, the core of generative AI, is already infused into EDB Postgres AI providing a platform for a range of practical and effective AI/ML solutions. + +### EDB Postgres AI - Platforms and Tools: + +* Postgres’s extensions are a source of its power and popularity, and are one of the categories that fall within this element of EDB Postgres AI. + +* Extensions sit alongside existing applications like Postgres Enterprise Manager, Barman, and Query Advisor as tools that allow you to leverage Postgres’s capabilities. + +* Also within this element are EDB’s Migration tools, Migration Toolkit and Migration Portal. The Migration Portal is among the first EDB tools to include embedded AI with an AI copilot that can assist users in developing migration strategies. + diff --git a/advocacy_docs/edb_postgres_ai/overview/features.mdx b/advocacy_docs/edb_postgres_ai/overview/features.mdx new file mode 100644 index 00000000000..c6817877d57 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/overview/features.mdx @@ -0,0 +1,11 @@ +--- +title: EDB Postgres AI Overview - Features +navTitle: Features +description: The current features of EDB Postgres AI, element by element. +--- + +What are the features of EDB Postgres AI? This section provides a detailed look at the features of EDB Postgres AI, element by element. + +* _ +* _ +* _ diff --git a/advocacy_docs/edb_postgres_ai/overview/guide.mdx b/advocacy_docs/edb_postgres_ai/overview/guide.mdx new file mode 100644 index 00000000000..d2d08853c95 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/overview/guide.mdx @@ -0,0 +1,6 @@ +--- +title: EDB Postgres AI Overview - Guide +navTitle: Guide +description: What do you want to use EDB Postgres AI for? Start navigating the documentation here. +--- + diff --git a/advocacy_docs/edb_postgres_ai/overview/index.mdx b/advocacy_docs/edb_postgres_ai/overview/index.mdx new file mode 100644 index 00000000000..5cac132bd29 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/overview/index.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Overview +navTitle: Overview +indexCards: simple +--- + +The overview of EDB Postgres AI provides a wide-angle view of the product and its capabilities. This section includes the following: + + + diff --git a/advocacy_docs/edb_postgres_ai/tools/backup.mdx b/advocacy_docs/edb_postgres_ai/tools/backup.mdx new file mode 100644 index 00000000000..023a7f66927 --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/tools/backup.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Tools - Backup and Recovery +navTitle: Backup and Recovery +description: The backup and recovery tools available in EDB Postgres AI Tools +--- + +Barman is a tool for managing backup and recovery of PostgreSQL databases. It is designed for business critical databases and provides features such as backup catalogues, incremental backup, retention policies, and remote recovery. + +Barman is integrated with EDB Postgres AI and can be used to manage backups of your EDB Postgres AI databases. + diff --git a/advocacy_docs/edb_postgres_ai/tools/index.mdx b/advocacy_docs/edb_postgres_ai/tools/index.mdx new file mode 100644 index 00000000000..26987f3948d --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/tools/index.mdx @@ -0,0 +1,16 @@ +--- +title: EDB Postgres AI - Tools +navTitle: Tools +indexCards: simple +navigation: +- migration-and-ai +- management +- backup +--- + +EDB Postgres AI Tools + +Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. + +_EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered._ + diff --git a/advocacy_docs/edb_postgres_ai/tools/management.mdx b/advocacy_docs/edb_postgres_ai/tools/management.mdx new file mode 100644 index 00000000000..683640daeab --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/tools/management.mdx @@ -0,0 +1,11 @@ +--- +title: EDB Postgres AI Tools - Management +navTitle: Management +description: An introduction to the management tools of EDB Postgres AI such as Postgres Enterprise Manager. +--- + +An introduction to the management tools of EDB Postgres AI such as Postgres Enterprise Manager. + +PEM is a comprehensive management tool for EDB Postgres Advanced Server and PostgreSQL databases. PEM provides database administrators with a graphical view of the server, allowing them to easily monitor and manage their databases. PEM also provides tools for database design, monitoring, and tuning, as well as tools for managing database objects, users, and roles. + +PEM is a web-based application that can be accessed from any web browser. PEM provides a single pane of glass for managing multiple database servers, allowing administrators to easily monitor and manage their databases from a centralized location. \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx b/advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx new file mode 100644 index 00000000000..662be27c24d --- /dev/null +++ b/advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx @@ -0,0 +1,13 @@ +--- +title: EDB Postgres AI Tools - Migration and AI +navTitle: Migration and AI +description: The Migration offering of EDB Postgres AI Tools includes an innovative migration copilot. +--- + +EDB Postgres AI Tools Migration Portal already offeres an AI copilot to assist users who are migration their databases to EDB Postgres. The AI copilot is a chatbot that helps users with the migration process. The AI copilot is designed to help users with the following tasks: + +- **Migration Assessment**: The AI copilot can help users assess their migration readiness. +- **Migration Planning**: The AI copilot can help users plan their migration. +- **General Migration Assistance**: The AI copilot can help users with general migration questions. + +The AI copilot is designed to be user-friendly and easy to use. Users can interact with the AI copilot using natural language. The AI copilot is also designed to be context-aware, so it can provide users with relevant information based on the context of the conversation. \ No newline at end of file diff --git a/build-sources.json b/build-sources.json index a2c27ddf443..008148c6c6b 100644 --- a/build-sources.json +++ b/build-sources.json @@ -34,5 +34,6 @@ "slony": true, "tde": true, "tpa": true, - "wait_states": true + "wait_states": true, + "edb_postgres_ai": true } diff --git a/gatsby-config.js b/gatsby-config.js index 5c9ff089fb5..5622fbcedc8 100644 --- a/gatsby-config.js +++ b/gatsby-config.js @@ -103,6 +103,10 @@ const sourceToPluginConfig = { name: "wait_states", path: "advocacy_docs/pg_extensions/wait_states", }, + edb_postgres_ai: { + name: "edb_postgres_ai", + path: "advocacy_docs/edb_postgres_ai", + }, }; const externalSourcePlugins = () => { diff --git a/src/components/icon/iconNames.js b/src/components/icon/iconNames.js index 6c216ce9dcf..0626110cef1 100644 --- a/src/components/icon/iconNames.js +++ b/src/components/icon/iconNames.js @@ -279,5 +279,8 @@ const iconNames = { NOT_FOUND: "color/NotFound", SERVER_ERROR: "color/ServerError", SUN: "color/Sun", + EDB_POSTGRES_AI_LOOP: "edb_postgres_ai/LoopColor", + EDB_POSTGRES_AI_LOOP_BLACK: "edb_postgres_ai/LoopBlack", + EDB_POSTGRES_AI_LANDSCAPE_COLOR: "edb_postgres_ai/LandscapeColor", }; export default iconNames; diff --git a/src/components/icon/index.js b/src/components/icon/index.js index 7752bd4cce1..c971507e7bf 100644 --- a/src/components/icon/index.js +++ b/src/components/icon/index.js @@ -4,6 +4,7 @@ import iconNames from "./iconNames"; import * as defaultIcons from "@enterprisedb/icons"; import * as colorIcons from "@enterprisedb/icons/color"; import * as logosIcons from "@enterprisedb/icons/logos"; +import * as ebd_postgres_aiIcons from "@enterprisedb/icons/edb_logos"; function IconContainer({ circle, @@ -42,6 +43,8 @@ const Icon = ({ category, name, ...props }) => { SelectedIcon = logosIcons[name]; } else if (category === "color") { SelectedIcon = colorIcons[name]; + } else if (category === "edb_postgres_ai") { + SelectedIcon = ebd_postgres_aiIcons[name]; } if (!SelectedIcon) SelectedIcon = "span"; diff --git a/src/components/left-nav.js b/src/components/left-nav.js index 194d8dfd7dc..1bffeb3d8dc 100644 --- a/src/components/left-nav.js +++ b/src/components/left-nav.js @@ -10,11 +10,18 @@ const productIcon = (path) => { }; const SectionHeading = ({ navTree, path, iconName }) => { + // if iconName starts with "edb_postgres_ai" then set the fill color to black + let myIconName = iconName || productIcon(path) || iconNames.DOTTED_BOX; + let className = "fill-orange me-3"; + if (myIconName && myIconName.startsWith("edb_postgres_ai")) { + className = "fill-black me-3"; + } + return (
  • diff --git a/src/components/logo.js b/src/components/logo.js index 9e3c69f5df5..da11a17d52a 100644 --- a/src/components/logo.js +++ b/src/components/logo.js @@ -1,5 +1,5 @@ import React from "react"; -import LogoSvg from "../../static/icons/edb-logo-disc-dark.svg"; +import LogoSvg from "../../static/icons/edb_landscape_color_grey.svg"; function Logo({ className, width, height }) { return ; diff --git a/src/constants/products.js b/src/constants/products.js index d0779fce3f6..9372e9aadbb 100644 --- a/src/constants/products.js +++ b/src/constants/products.js @@ -96,4 +96,8 @@ export const products = { tde: { name: "Transparent Data Encryption", iconName: IconNames.SECURITY }, tpa: { name: "Trusted Postgres Architect", iconName: IconNames.INSTANCES }, wait_states: { name: "EDB Wait States", iconName: IconNames.POSTGRESQL }, + edb_postgres_ai: { + name: "EDB Postgres AI", + iconName: IconNames.EDB_POSTGRES_AI_LOOP_BLACK, + }, }; diff --git a/src/pages/index.js b/src/pages/index.js index 8df73d76fdc..1e4f20ee050 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -25,6 +25,53 @@ const IndexCard = ({ iconName, headingText, children }) => (
    • ); +const BannerCard = ({ iconName, headingText, children }) => ( +
    +
    +
    +
    + +

    {headingText}

    +
    +
    {children}
    +
    +
    +
    +); + +const BannerSubCard = ({ iconName, headingText, to, children }) => ( +
    +
    +
    +
    + + +

    {headingText}

    + +
    +
    +
    {children}
    +
    +
    +
    +
    +); +const BannerCardLink = ({ to, className, children }) => ( + + {children} + +); + const IndexCardLink = ({ to, className, children }) => (
  • @@ -114,6 +161,103 @@ const Page = () => { {/* Sign Post */} + + + + Concepts + + + Guide + + + Features + + + + + Getting Started + + + Estate + + + Agent + + + + + Cloud Service + + + On Premises + + + High Availability + + + + + Getting Started + + + + Lakehouse + + + Analytics + + + + + Overview + + + Databases and AI + + + Postgres and AI + + + + + Migration and AI + + + Management + + + Backup and Recovery + + + +
    diff --git a/static/icons/edb_landscape_color_grey.svg b/static/icons/edb_landscape_color_grey.svg new file mode 100644 index 00000000000..97dee10d539 --- /dev/null +++ b/static/icons/edb_landscape_color_grey.svg @@ -0,0 +1,44 @@ + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 1fb4687696bed9fe7f2c5c951b232ce45006a81f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Mon, 20 May 2024 05:36:59 +0100 Subject: [PATCH 24/67] Remove placeholder nav --- src/pages/index.js | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/src/pages/index.js b/src/pages/index.js index 1e4f20ee050..1cbc54b0f5c 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -144,23 +144,6 @@ const Page = () => {
    - {/* Postgres AI nav goes here */} - -
    -

    Postgres AI nav goes here

    - Console - - Databases - - Analytics - - AI-ML - - Tools -
    - - {/* Sign Post */} - Date: Mon, 20 May 2024 09:31:16 -0700 Subject: [PATCH 25/67] Consolidate paths. Link product definition. Remove unnecessary configuration. --- .../ai-ml/databases-and-ai.mdx | 0 advocacy_docs/edb-postgres-ai/ai-ml/index.mdx | 15 ++++-- .../ai-ml/overview.mdx | 0 .../ai-ml/postgres-and-ai.mdx | 0 .../analytics/analytics.mdx | 0 .../analytics/getstarted.mdx | 0 .../edb-postgres-ai/analytics/index.mdx | 14 +++++- .../analytics/lakehouse.mdx | 0 .../console/agent.mdx | 0 .../console/estate.mdx | 0 .../console/getstarted.mdx | 0 .../edb-postgres-ai/console/index.mdx | 11 ++++- .../databases/cloudservice.mdx | 0 .../edb-postgres-ai/databases/index.mdx | 10 +++- .../databases/onprem.mdx | 0 .../databases/options.mdx | 0 advocacy_docs/edb-postgres-ai/index.mdx | 20 +++++--- .../overview/concepts.mdx | 0 .../overview/features.mdx | 0 .../overview/guide.mdx | 0 .../overview/index.mdx | 1 + .../tools/backup.mdx | 0 advocacy_docs/edb-postgres-ai/tools/index.mdx | 14 +++++- .../tools/management.mdx | 0 .../tools/migration-and-ai.mdx | 0 advocacy_docs/edb_postgres_ai/ai-ml/index.mdx | 13 ----- .../edb_postgres_ai/analytics/index.mdx | 14 ------ .../edb_postgres_ai/console/index.mdx | 13 ----- .../edb_postgres_ai/databases/index.mdx | 12 ----- advocacy_docs/edb_postgres_ai/index.mdx | 15 ------ advocacy_docs/edb_postgres_ai/tools/index.mdx | 16 ------- build-sources.json | 3 +- gatsby-config.js | 4 -- src/constants/products.js | 2 +- src/pages/index.js | 48 +++++++++---------- 35 files changed, 96 insertions(+), 129 deletions(-) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/ai-ml/databases-and-ai.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/ai-ml/overview.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/ai-ml/postgres-and-ai.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/analytics/analytics.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/analytics/getstarted.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/analytics/lakehouse.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/console/agent.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/console/estate.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/console/getstarted.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/databases/cloudservice.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/databases/onprem.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/databases/options.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/overview/concepts.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/overview/features.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/overview/guide.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/overview/index.mdx (93%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/tools/backup.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/tools/management.mdx (100%) rename advocacy_docs/{edb_postgres_ai => edb-postgres-ai}/tools/migration-and-ai.mdx (100%) delete mode 100644 advocacy_docs/edb_postgres_ai/ai-ml/index.mdx delete mode 100644 advocacy_docs/edb_postgres_ai/analytics/index.mdx delete mode 100644 advocacy_docs/edb_postgres_ai/console/index.mdx delete mode 100644 advocacy_docs/edb_postgres_ai/databases/index.mdx delete mode 100644 advocacy_docs/edb_postgres_ai/index.mdx delete mode 100644 advocacy_docs/edb_postgres_ai/tools/index.mdx diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/ai-ml/databases-and-ai.mdx rename to advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx index 073269a40e5..3be6994c1a0 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx @@ -1,5 +1,14 @@ --- -title: Postgres AI - AI-ML -navTitle: AI-ML -iconName: Learning +title: EDB Postgres AI - AI/ML +navTitle: AI/ML +indexCards: simple +iconName: BrainCircuit +navigation: +- overview +- databases-and-ai +- postgres-and-ai --- + +EDB Postgres AI AI/ML. + +_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/ai-ml/overview.mdx rename to advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/ai-ml/postgres-and-ai.mdx rename to advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx diff --git a/advocacy_docs/edb_postgres_ai/analytics/analytics.mdx b/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/analytics/analytics.mdx rename to advocacy_docs/edb-postgres-ai/analytics/analytics.mdx diff --git a/advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx b/advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/analytics/getstarted.mdx rename to advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index c20a3b53944..2a0a257aab6 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -1,5 +1,15 @@ --- -title: Postgres AI - Analytics +title: EDB Postgres AI Analytics navTitle: Analytics -iconName: CrystalBall +indexCards: simple +iconName: Improve +navigation: +- getstarted +- analytics +- lakehouse --- + +Introduction to EDB Postgres AI Analytics. + +_EDB Postgres AI Analytics provides the tools needed to gather, analyze, and visualize data. It includes a variety of tools and features that help you to analyze data and make data-driven decisions. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ + diff --git a/advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/analytics/lakehouse.mdx rename to advocacy_docs/edb-postgres-ai/analytics/lakehouse.mdx diff --git a/advocacy_docs/edb_postgres_ai/console/agent.mdx b/advocacy_docs/edb-postgres-ai/console/agent.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/console/agent.mdx rename to advocacy_docs/edb-postgres-ai/console/agent.mdx diff --git a/advocacy_docs/edb_postgres_ai/console/estate.mdx b/advocacy_docs/edb-postgres-ai/console/estate.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/console/estate.mdx rename to advocacy_docs/edb-postgres-ai/console/estate.mdx diff --git a/advocacy_docs/edb_postgres_ai/console/getstarted.mdx b/advocacy_docs/edb-postgres-ai/console/getstarted.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/console/getstarted.mdx rename to advocacy_docs/edb-postgres-ai/console/getstarted.mdx diff --git a/advocacy_docs/edb-postgres-ai/console/index.mdx b/advocacy_docs/edb-postgres-ai/console/index.mdx index af37ac0f826..608be3c9b67 100644 --- a/advocacy_docs/edb-postgres-ai/console/index.mdx +++ b/advocacy_docs/edb-postgres-ai/console/index.mdx @@ -1,5 +1,14 @@ --- -title: Postgres AI - Console +title: EDB Postgres AI Console navTitle: Console +indexCards: simple iconName: Control +navigation: +- getstarted +- estate +- agent --- + +Introduction to the EDB Postgres AI Console + +_The EDB Postgres AI Console is a web-based interface that allows you to manage your EDB Postgres AI environment. The console provides a user-friendly interface for managing your estate, including the remote agents and databases that run on your servers._ diff --git a/advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx b/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/databases/cloudservice.mdx rename to advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx diff --git a/advocacy_docs/edb-postgres-ai/databases/index.mdx b/advocacy_docs/edb-postgres-ai/databases/index.mdx index 154e2265087..aa56a6733d1 100644 --- a/advocacy_docs/edb-postgres-ai/databases/index.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/index.mdx @@ -1,5 +1,13 @@ --- -title: Postgres AI - Databases +title: EDB Postgres AI Databases navTitle: Databases +indexCards: simple iconName: Database --- + +Introduction to the EDB Postgres AI Databases and the EDB Postgres AI Cloud Service + +_The EDB Postgres AI Databases are a set of databases that are optimized for machine learning and artificial intelligence workloads. The databases are designed to provide high performance and scalability for machine learning and artificial intelligence workloads. The EDB Postgres AI Databases are available as part of the EDB Postgres AI Cloud Service._ + + + diff --git a/advocacy_docs/edb_postgres_ai/databases/onprem.mdx b/advocacy_docs/edb-postgres-ai/databases/onprem.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/databases/onprem.mdx rename to advocacy_docs/edb-postgres-ai/databases/onprem.mdx diff --git a/advocacy_docs/edb_postgres_ai/databases/options.mdx b/advocacy_docs/edb-postgres-ai/databases/options.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/databases/options.mdx rename to advocacy_docs/edb-postgres-ai/databases/options.mdx diff --git a/advocacy_docs/edb-postgres-ai/index.mdx b/advocacy_docs/edb-postgres-ai/index.mdx index 7f5d29717bc..708c7faffbd 100644 --- a/advocacy_docs/edb-postgres-ai/index.mdx +++ b/advocacy_docs/edb-postgres-ai/index.mdx @@ -1,12 +1,18 @@ --- -title: Postgres AI +title: EDB Postgres AI Home +navTitle: Home directoryDefaults: - iconName: BrainCircuit + product: "EDB Postgres AI" + iconName: edb_postgres_ai/LoopBlack indexCards: full navigation: - - console - - databases - - analytics - - ai-ml - - tools +- overview +- console +- databases +- analytics +- ai-ml +- tools --- + +The future home of all EDB Postgres AI documentation. Stay tuned! + diff --git a/advocacy_docs/edb_postgres_ai/overview/concepts.mdx b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/overview/concepts.mdx rename to advocacy_docs/edb-postgres-ai/overview/concepts.mdx diff --git a/advocacy_docs/edb_postgres_ai/overview/features.mdx b/advocacy_docs/edb-postgres-ai/overview/features.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/overview/features.mdx rename to advocacy_docs/edb-postgres-ai/overview/features.mdx diff --git a/advocacy_docs/edb_postgres_ai/overview/guide.mdx b/advocacy_docs/edb-postgres-ai/overview/guide.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/overview/guide.mdx rename to advocacy_docs/edb-postgres-ai/overview/guide.mdx diff --git a/advocacy_docs/edb_postgres_ai/overview/index.mdx b/advocacy_docs/edb-postgres-ai/overview/index.mdx similarity index 93% rename from advocacy_docs/edb_postgres_ai/overview/index.mdx rename to advocacy_docs/edb-postgres-ai/overview/index.mdx index 5cac132bd29..c425177a694 100644 --- a/advocacy_docs/edb_postgres_ai/overview/index.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/index.mdx @@ -2,6 +2,7 @@ title: EDB Postgres AI Overview navTitle: Overview indexCards: simple +iconName: Earth --- The overview of EDB Postgres AI provides a wide-angle view of the product and its capabilities. This section includes the following: diff --git a/advocacy_docs/edb_postgres_ai/tools/backup.mdx b/advocacy_docs/edb-postgres-ai/tools/backup.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/tools/backup.mdx rename to advocacy_docs/edb-postgres-ai/tools/backup.mdx diff --git a/advocacy_docs/edb-postgres-ai/tools/index.mdx b/advocacy_docs/edb-postgres-ai/tools/index.mdx index 8ddd8ca4d65..3c4b4c8779c 100644 --- a/advocacy_docs/edb-postgres-ai/tools/index.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/index.mdx @@ -1,5 +1,17 @@ --- -title: Postgres AI - Tools +title: EDB Postgres AI - Tools navTitle: Tools +indexCards: simple iconName: Toolbox +navigation: +- migration-and-ai +- management +- backup --- + +EDB Postgres AI Tools + +Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. + +_EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered._ + diff --git a/advocacy_docs/edb_postgres_ai/tools/management.mdx b/advocacy_docs/edb-postgres-ai/tools/management.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/tools/management.mdx rename to advocacy_docs/edb-postgres-ai/tools/management.mdx diff --git a/advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx b/advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx similarity index 100% rename from advocacy_docs/edb_postgres_ai/tools/migration-and-ai.mdx rename to advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx diff --git a/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx b/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx deleted file mode 100644 index fe53c8645bf..00000000000 --- a/advocacy_docs/edb_postgres_ai/ai-ml/index.mdx +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: EDB Postgres AI - AI/ML -navTitle: AI/ML -indexCards: simple -navigation: -- overview -- databases-and-ai -- postgres-and-ai ---- - -EDB Postgres AI AI/ML. - -_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ \ No newline at end of file diff --git a/advocacy_docs/edb_postgres_ai/analytics/index.mdx b/advocacy_docs/edb_postgres_ai/analytics/index.mdx deleted file mode 100644 index b286cad418c..00000000000 --- a/advocacy_docs/edb_postgres_ai/analytics/index.mdx +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: EDB Postgres AI Analytics -navTitle: Analytics -indexCards: simple -navigation: -- getstarted -- analytics -- lakehouse ---- - -Introduction to EDB Postgres AI Analytics. - -_EDB Postgres AI Analytics provides the tools needed to gather, analyze, and visualize data. It includes a variety of tools and features that help you to analyze data and make data-driven decisions. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ - diff --git a/advocacy_docs/edb_postgres_ai/console/index.mdx b/advocacy_docs/edb_postgres_ai/console/index.mdx deleted file mode 100644 index 84e05c0c506..00000000000 --- a/advocacy_docs/edb_postgres_ai/console/index.mdx +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: EDB Postgres AI Console -navTitle: Console -indexCards: simple -navigation: -- getstarted -- estate -- agent ---- - -Introduction to the EDB Postgres AI Console - -_The EDB Postgres AI Console is a web-based interface that allows you to manage your EDB Postgres AI environment. The console provides a user-friendly interface for managing your estate, including the remote agents and databases that run on your servers._ diff --git a/advocacy_docs/edb_postgres_ai/databases/index.mdx b/advocacy_docs/edb_postgres_ai/databases/index.mdx deleted file mode 100644 index f9878718643..00000000000 --- a/advocacy_docs/edb_postgres_ai/databases/index.mdx +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: EDB Postgres AI Databases -navTitle: Databases -indexCards: simple ---- - -Introduction to the EDB Postgres AI Databases and the EDB Postgres AI Cloud Service - -_The EDB Postgres AI Databases are a set of databases that are optimized for machine learning and artificial intelligence workloads. The databases are designed to provide high performance and scalability for machine learning and artificial intelligence workloads. The EDB Postgres AI Databases are available as part of the EDB Postgres AI Cloud Service._ - - - diff --git a/advocacy_docs/edb_postgres_ai/index.mdx b/advocacy_docs/edb_postgres_ai/index.mdx deleted file mode 100644 index 4b9aaaa93ee..00000000000 --- a/advocacy_docs/edb_postgres_ai/index.mdx +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: EDB Postgres AI Home -navTitle: Home -indexCards: full -navigation: -- overview -- console -- databases -- analytics -- ai-ml -- tools ---- - -The future home of all EDB Postgres AI documentation. Stay tuned! - diff --git a/advocacy_docs/edb_postgres_ai/tools/index.mdx b/advocacy_docs/edb_postgres_ai/tools/index.mdx deleted file mode 100644 index 26987f3948d..00000000000 --- a/advocacy_docs/edb_postgres_ai/tools/index.mdx +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: EDB Postgres AI - Tools -navTitle: Tools -indexCards: simple -navigation: -- migration-and-ai -- management -- backup ---- - -EDB Postgres AI Tools - -Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. - -_EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered._ - diff --git a/build-sources.json b/build-sources.json index 008148c6c6b..a2c27ddf443 100644 --- a/build-sources.json +++ b/build-sources.json @@ -34,6 +34,5 @@ "slony": true, "tde": true, "tpa": true, - "wait_states": true, - "edb_postgres_ai": true + "wait_states": true } diff --git a/gatsby-config.js b/gatsby-config.js index 5622fbcedc8..5c9ff089fb5 100644 --- a/gatsby-config.js +++ b/gatsby-config.js @@ -103,10 +103,6 @@ const sourceToPluginConfig = { name: "wait_states", path: "advocacy_docs/pg_extensions/wait_states", }, - edb_postgres_ai: { - name: "edb_postgres_ai", - path: "advocacy_docs/edb_postgres_ai", - }, }; const externalSourcePlugins = () => { diff --git a/src/constants/products.js b/src/constants/products.js index 9372e9aadbb..ce5e7700366 100644 --- a/src/constants/products.js +++ b/src/constants/products.js @@ -96,7 +96,7 @@ export const products = { tde: { name: "Transparent Data Encryption", iconName: IconNames.SECURITY }, tpa: { name: "Trusted Postgres Architect", iconName: IconNames.INSTANCES }, wait_states: { name: "EDB Wait States", iconName: IconNames.POSTGRESQL }, - edb_postgres_ai: { + "EDB Postgres AI": { name: "EDB Postgres AI", iconName: IconNames.EDB_POSTGRES_AI_LOOP_BLACK, }, diff --git a/src/pages/index.js b/src/pages/index.js index 1cbc54b0f5c..e7103b1f415 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -151,91 +151,91 @@ const Page = () => { - + Concepts - + Guide - + Features - + Getting Started - + Estate - + Agent - + Cloud Service - + On Premises - + High Availability - + Getting Started - + Lakehouse - + Analytics - + Overview - + Databases and AI - + Postgres and AI - + Migration and AI - + Management - + Backup and Recovery From f965cc39a639bafcdcc817eab9c255525fb2946e Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 20 May 2024 10:29:22 +0100 Subject: [PATCH 26/67] Some text build for the AI section concepts added reference section to console. Signed-off-by: Dj Walker-Morgan --- .../ai-ml/databases-and-ai.mdx | 2 +- .../edb-postgres-ai/ai-ml/overview.mdx | 26 +++++++++++++++++-- .../edb-postgres-ai/console/index.mdx | 1 + .../console/reference/index.mdx | 6 +++++ 4 files changed, 32 insertions(+), 3 deletions(-) create mode 100644 advocacy_docs/edb-postgres-ai/console/reference/index.mdx diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx index 9cb0c30858e..86cf7a76b59 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx @@ -4,4 +4,4 @@ navTitle: Databases and AI description: An introduction to the intersection of databases and AI/ML with EDB Postgres AI. --- -_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!__ +_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!_ diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx index 6149bda5fa4..67660fa9e32 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx @@ -4,8 +4,30 @@ navTitle: Overview description: Where to start with EDB Postgres AI AI/ML, from creating innovative models to putting them into production. --- -Where to start with EDB Postgres AI AI/ML, from creating innovative models to putting them into production. +At the heart of EDB Postgres AI is the EDB Postgres AI database. This builds on Postgres's flexibility and extends its capability to include store the vector data of embeddings. + +## What is vector data? + +To be able to deterimine similarity between things, we need to be able to represent them in a way that allows us to compare them. This is where vector data comes in, as it reduces the complexity of an thing to a set of numbers that can be compared. Those numbers could be derived from a set of images of the thing, a set of words about it or even a set of sounds. Each number exists on a dimension of the vector and vectors can have as many dimensions at they need to fully repreent a thing. + +## What are embeddings? + +These sets of values for the dimensions are called embeddings. Once we have embeddings for real world object, or images or text, we can compare them to see how similar they are across all the aspects represented by the embeddings dimensions. This comparison a core operation performed by machine learning and artificial intelligence models to work out connections between things. + +## How are embeddings created? + +Embeddings can be created manually, by experts who understand the thing to be represented. This is expensive and doesn't scale. The other option is for it to be created automatically by machine learning models, which can be initially trained by experts. These models are then trained on a set of data that represents the thing to be embedded, and then the model is used to generate the embeddings. The model is tuned by adjusting the values of the embeddings until the model can predict the thing accurately. + +## How are embeddings stored in a database? + +Embeddings, as sets of numbers, are not easily stored in a traditional relational database. They are best stored in a database that is designed to handle them. This is where EDB Postgres AI comes in. Building on the popular PG/Vector extension, EDB Postgres AI extends the capabilities of Postgres to handle vector data, and then provides the tools and utilties you need to build and deploy machine learning and artificial intelligence models. + +## What can I do with EDB Postgres AI? + +In the coming months, we'll be adding more content to this site to help you get started with EDB Postgres AI. We'll be covering how to create embeddings, how to train models and how to deploy them. We'll also be covering how to use the models in your applications, and how to monitor and manage them. We'll also be covering how to scale your models and how to use them in a production environment. + + + -_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ diff --git a/advocacy_docs/edb-postgres-ai/console/index.mdx b/advocacy_docs/edb-postgres-ai/console/index.mdx index 608be3c9b67..7e4bce5d9f7 100644 --- a/advocacy_docs/edb-postgres-ai/console/index.mdx +++ b/advocacy_docs/edb-postgres-ai/console/index.mdx @@ -7,6 +7,7 @@ navigation: - getstarted - estate - agent +- reference --- Introduction to the EDB Postgres AI Console diff --git a/advocacy_docs/edb-postgres-ai/console/reference/index.mdx b/advocacy_docs/edb-postgres-ai/console/reference/index.mdx new file mode 100644 index 00000000000..4b053dc9697 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/reference/index.mdx @@ -0,0 +1,6 @@ +--- +title: EDB Postgres AI Console Reference +navTitle: Reference +description: Reference documentation for the EDB Postgres AI Console. +--- + From 24cd60b1c6109340d0453f0f5036b104e3063596 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 20 May 2024 18:05:54 +0100 Subject: [PATCH 27/67] Reorg for edb-postgres-ai Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/analytics/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 2a0a257aab6..4975d457302 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -2,7 +2,7 @@ title: EDB Postgres AI Analytics navTitle: Analytics indexCards: simple -iconName: Improve +iconName: CrystalBall navigation: - getstarted - analytics From 14e011e71464c64211dd808248d511fd682af5af Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 20 May 2024 18:16:48 +0100 Subject: [PATCH 28/67] Fix collapsing and analytics icon Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/analytics/index.mdx | 2 +- src/pages/index.js | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 4975d457302..2a0a257aab6 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -2,7 +2,7 @@ title: EDB Postgres AI Analytics navTitle: Analytics indexCards: simple -iconName: CrystalBall +iconName: Improve navigation: - getstarted - analytics diff --git a/src/pages/index.js b/src/pages/index.js index e7103b1f415..0a1fcca3d98 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -26,7 +26,7 @@ const IndexCard = ({ iconName, headingText, children }) => ( ); const BannerCard = ({ iconName, headingText, children }) => ( -
    +
    From 7caecc56978e36d3f5f16679e6ae294d9a56ca85 Mon Sep 17 00:00:00 2001 From: gvasquezvargas Date: Wed, 22 May 2024 12:36:06 +0200 Subject: [PATCH 29/67] Indentation error + test --- advocacy_docs/edb-postgres-ai/overview/concepts.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx index e434bf4dc3d..a043d683c24 100644 --- a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx @@ -36,8 +36,9 @@ EDB Postgres AI is composed of multiple elements which come together to deliver * Filtering out the data noise and revealing insights and value, Analytics brings both structured relational data in Postgres and unstructured data in object storage together for exploration. At the heart of Analytics is a custom built store for this data: -* ### EDB Postgres AI - Lakehouse: - * Built to bring structured and unstructured data together, Lakehouse supports numerous formats to bring your data in from the cold, ready to be analyzed. +### EDB Postgres AI - Lakehouse: + +* Built to bring structured and unstructured data together, Lakehouse supports numerous formats to bring your data in from the cold, ready to be analyzed. ### EDB Postgres AI - AI/ML: From a470d5619b3fdfbebab6032d70b92f3b579e47db Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Wed, 15 May 2024 15:37:49 -0400 Subject: [PATCH 30/67] Initial draft of self-managed monitoring. --- .../databases/pai-agent-self-managed.mdx | 304 ++++++++++++++++++ 1 file changed, 304 insertions(+) create mode 100644 advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx diff --git a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx new file mode 100644 index 00000000000..f7a450d9707 --- /dev/null +++ b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx @@ -0,0 +1,304 @@ +--- +title: Installing EDB Postgres AI agent for monitoring self-managed Postgres deployments +navTitle: Monitoring self-managed deployments +deepToC: true +--- + +To monitor a self-managed Postgres deployment using EDB Postgres AI, you need a machine agent configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. + +This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before going into the details of how to configure the EDB Postgres AI agent. There are also optional instructions for running the EDB Postgres AI agent as a service. + +## Setting up a machine user for estate ingestion in EDB Postgres AI Console + +1. In the EDB Postgres AI Console, using your avatar's dropdown, select the **User Management** page. + +2. Select the **Add New User** button. + +3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally select the **Add User** button. + +4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. + +5. Next, go to the project of the database you want to monitor, select the **Users** tab, and locate the machine user you just created. + +6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. + +7. In the **Assign Project Roles** modal, select "estate ingester" and then select the **Submit** button. + +8. Your machine user is ready for estate ingestions from the EDB Postgres AI agent. + +## Installing, configuring, testing, and running the Postgres AI agent + +The following steps walk you through how to install and configure the EDB Postgres AI agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. + +1. Establish a connection between your system and the Cloudsmith repository for EDB Postgres AI. + + First, find your EnterpriseDB Repos 2.0 token [here](https://www.enterprisedb.com/repos-downloads). + + Next, download and run a script to configure your system to be able to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token: + + For RHEL-like or SLES: + + ``` + curl -1sLf 'https://downloads.enterprisedb.com//dev/setup.rpm.sh' | sudo -E bash + ``` + + For Debian or Ubuntu: + + ``` + curl -1sLf 'https://downloads.enterprisedb.com//dev/setup.deb.sh' | sudo -E bash + ``` + +2. Install the `beacon-agent` package: + + For RHEL-like: + + ``` + dnf install beacon-agent + ``` + + Or if dnf isn't available: + + ``` + yum install beacon-agent + ``` + + For SLES: + + ``` + zypper install beacon-agent + ``` + + For Debian or Ubuntu: + + ``` + apt install beacon-agent + ``` + +3. Configure EDB Postgres AI agent and database connections. + + First, create a beacon config in your home directory: + + ``` + mkdir ${HOME}/.beacon + ``` + + Next, configure the beacon agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: + + ``` + export BEACON_AGENT_ACCESS_KEY= + export BEACON_AGENT_PROJECT_ID= + beacon-agent setup -file="$HOME/.beacon/beacon_agent.yaml" + ``` + + These commands set up the necessary parameters for the Beacon agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. + + During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for the Beacon agent to communicate with the Beacon service and register successfully. + + If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your Beacon project. + + +4. For security best practices, create a `DSN` environmental variable for each database (`DSN1`, `DSN2`, and so on) you want to monitor and set each to the correct corresponding dsn. + + ``` + export DSN1= + ``` + +5. Open your `$HOME/.beacon/beacon_agent.yaml` configuration file and add an entry for each database you want to connect to, including the environmental variable for each database's dsn. + + Entries under `databases` utilize the following format: + + ``` + databases: + : + dsn: "$DSN1" + tags: + - "" + - "" + : + dsn: "$DSN2" + tags: + - "" + - "" + ``` + + Here is an example with a database named `john_laptop_0`: +``` +agent: + access_key: $BEACON_AGENT_ACCESS_KEY + access_key_grpc_header: "x-access-key" + batch_size: 100 + beacon_server: "beacon.biganimal.com:443" + feature_flag_interval: 10m0s + project_id: "" + providers: + - "onprem" +provider: + onprem: + databases: + john_laptop_0: + dsn: “$DSN” + tags: + - "foo" + - "bar" + host: + resource_id: "Johns-MBP.lan" + tags: [] + poll_interval: 5m0s +``` + +6. Test the EDB Postgres AI agent locally. + + As an initial smoke test of the agent, it can send the ingestions that it would send back to the beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. + + You can run the agent in stdout mode by modifying the `beacon_agent.yaml` file generated previously to have an `agent.beacon_server` value of `"stdout"`. A truncated example of this would be: + + ``` + agent: + beacon_server: "stdout" + ``` + + Next, run the agent in this mode using the following command: + + ``` + beacon-agent + ``` + + You should see output similar to the following: + +``` +{"level":"debug","data":"$BEACON_AGENT_ACCESS_KEY","time":"2024-05-08T18:40:34Z","message":"expanding environment variable in configuration"} +{"level":"info","path":"/healthz","time":1715193634,"msg":"serving liveness probe"} +{"level":"info","path":"/readyz","time":1715193634,"msg":"serving readiness probe"} +{"level":"info","version":"v1.51.0-snapshot8986075626.97.1.166215e","time":1715193634,"msg":"starting beacon agent"} +{"level":"info","spiffe_enabled":false,"time":1715193634,"msg":"configuring tls"} +{"level":"info","server":"stdout","time":1715193634,"msg":"connecting to beacon service"} +{"level":"info","address":":8081","time":1715193634,"msg":"starting probe server"} +{"level":"info","target":"stdout","time":1715193634,"msg":"connected to beacon server"} +{"level":"info","time":1715193634,"msg":"verifying connection to beacon server"} +{"level":"info","project":"echo","time":1715193634,"msg":"verified connection to beacon server"} +{"level":"info","interval":"10m0s","time":1715193634,"msg":"loading feature flags periodically"} +{"level":"info","time":1715193634,"msg":"fetching feature flags"} +{"level":"info","feature_flags":{"echo_flag":"test","second_flag":false},"time":1715193634,"msg":"loaded feature flags"} +{"level":"info","id":"onprem","time":1715193634,"msg":"starting provider"} +{"level":"info","disable_partitioning":false,"batch_size":100,"interval":"10s","time":1715193634,"msg":"starting batch exporter"} +{"level":"info","provider_id":"onprem","time":1715193634,"msg":"registering ingestion worker in pool"} +{"level":"info","provider":"onprem","time":1715193634,"msg":"starting provider worker"} +{"level":"info","ingestions":[{"version":"v0.1.0","type":"onprem/host","id":"ip-10-0-128-121","metadata":{"Data":{"OnPremHostMetadata":{"hostname":"ip-10-0-128-121","operating_system":"linux","platform":"ubuntu","platform_family":"debian","platform_version":"22.04","cpu_limit":1}}}}],"time":1715193934,"msg":"sending ingestions via log client (not actually sending)"} +{"level":"info","successful_ingestions":1,"failed_ingestions":0,"time":1715193934,"msg":"exported ingestions"} +``` + +!!! Note +The message in the second to last line of the logs above specifies that we are using the log client and not actually sending ingestions. +!!! + +7. Run the Beacon agent. + + In the `beacon_agent.yaml` file, replace `"stdout"` in `agent:beacon_server` with `"beacon.biganimal.com:443"`: + + ``` + agent: + beacon_server: "beacon.biganimal.com:443" + + ``` + +8. Follow the logs to monitor the ingestions. + +## Running Postgres AI agent as a service + +To have EDB Postgres AI agent run automatically on startup and restart after error, you need to make it a service. + +!!! Note +Future versions of the agent package may set this up automatically. +!!! + +What follows is an example of how to run EDB Postgres AI agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running the Beacon agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed on a Ubuntu 22.04 machine. + +1. Because, in this example, we are running the agent on the same server as the Postgres instance we're monitoring, it's faster and more secure to have the EDB Postgres AI agent use Postgres local auth rather than set up password auth over TCP/IP. + + !!! Note + In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production you'd want to use a minimally privileged user created specifically for the purposes of running the Beacon agent on the server. + !!! + + To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: + + ``` + sudo su postgres + psql -c 'CREATE USER ubuntu' + exit + ``` + + To complete the setup for local authentication with Postgres, you need to ensure your `pg_hba.conf` file is configured to allow Unix-domain socket connections. Please verify or update the following line in your `pg_hba.conf` file: + + ``` + local all all peer + ``` + + This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). + + Now Postgres is configured to allow local authentication so that the Beacon agent can access it as a service. + +2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: + +``` +sudo vi /etc/systemd/beacon-agent.service +``` + +3. Populate the new file as follows, specifying the BEACON_AGENT_ACCESS_KEY, then save and exit the editor: + +``` +[Unit] + +Description=The Postgres AI Agent + +# After networking because we need that + +After=network.target + +[Service] + +# Simple services don't do any forking / background nonsense + +Type=simple + +# User with which to run the service + +User=ubuntu + +# Set the working directory for the application + +WorkingDirectory=/home/ubuntu/ + +Environment='BEACON_AGENT_ACCESS_KEY=' + +# Command to run the application + +ExecStart=/usr/local/bin/beacon-agent + +# Restart policy, only on failure + +Restart=on-failure + +RestartSec=60 + +[Install] + +# Start the service before we get to multi-user mode + +WantedBy=multi-user.target +``` + +4. Run the following commands to reload, enable, and start your new service: + +``` +sudo systemctl daemon-reload +sudo systemctl enable beacon-agent.service +sudo systemctl start beacon-agent.service +``` + +5. Your agent should now be running as a service. Check on the logs by running the following command: + +``` +journalctl -u beacon-agent.service +``` + + From c0a2a0127c61840db22ca28bef6c6a634bc70c1f Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Thu, 16 May 2024 14:34:03 -0400 Subject: [PATCH 31/67] Fixed some references to Beacon. --- .../databases/pai-agent-self-managed.mdx | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx index f7a450d9707..b327c1de9e7 100644 --- a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx @@ -1,12 +1,12 @@ --- -title: Installing EDB Postgres AI agent for monitoring self-managed Postgres deployments +title: Installing EDB Postgres AI Agent for monitoring self-managed Postgres deployments navTitle: Monitoring self-managed deployments deepToC: true --- -To monitor a self-managed Postgres deployment using EDB Postgres AI, you need a machine agent configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. +To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine agent configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. -This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before going into the details of how to configure the EDB Postgres AI agent. There are also optional instructions for running the EDB Postgres AI agent as a service. +This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring the EDB Postgres AI Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. ## Setting up a machine user for estate ingestion in EDB Postgres AI Console @@ -18,15 +18,15 @@ This guide walks through the procedure for setting up a machine user in the EDB 4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. -5. Next, go to the project of the database you want to monitor, select the **Users** tab, and locate the machine user you just created. +5. Next, go to the project from which you want to monitor the database, select the **Users** tab, and locate the machine user you just created. 6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. -7. In the **Assign Project Roles** modal, select "estate ingester" and then select the **Submit** button. +7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. -8. Your machine user is ready for estate ingestions from the EDB Postgres AI agent. +8. Your machine user is ready for estate ingestions from the EDB Postgres AI Agent. -## Installing, configuring, testing, and running the Postgres AI agent +## Installing, configuring, testing, and running the EDB Postgres AI agent The following steps walk you through how to install and configure the EDB Postgres AI agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. @@ -34,7 +34,7 @@ The following steps walk you through how to install and configure the EDB Postgr First, find your EnterpriseDB Repos 2.0 token [here](https://www.enterprisedb.com/repos-downloads). - Next, download and run a script to configure your system to be able to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token: + Next, download and run a script to configure your system to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token: For RHEL-like or SLES: @@ -74,7 +74,7 @@ The following steps walk you through how to install and configure the EDB Postgr apt install beacon-agent ``` -3. Configure EDB Postgres AI agent and database connections. +3. Configure EDB Postgres AI Agent and database connections. First, create a beacon config in your home directory: @@ -92,9 +92,9 @@ The following steps walk you through how to install and configure the EDB Postgr These commands set up the necessary parameters for the Beacon agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. - During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for the Beacon agent to communicate with the Beacon service and register successfully. + During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for the EDB Postgres AI Agent to communicate with EDB Postgres AI and register successfully. - If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your Beacon project. + If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your EDB Postgres AI project. 4. For security best practices, create a `DSN` environmental variable for each database (`DSN1`, `DSN2`, and so on) you want to monitor and set each to the correct corresponding dsn. @@ -146,7 +146,7 @@ provider: poll_interval: 5m0s ``` -6. Test the EDB Postgres AI agent locally. +6. Test the EDB Postgres AI Agent locally. As an initial smoke test of the agent, it can send the ingestions that it would send back to the beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. @@ -211,9 +211,9 @@ To have EDB Postgres AI agent run automatically on startup and restart after err Future versions of the agent package may set this up automatically. !!! -What follows is an example of how to run EDB Postgres AI agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running the Beacon agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed on a Ubuntu 22.04 machine. +What follows is an example of how to run EDB Postgres AI agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running the Beacon agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed EDB Postgres AI Agent on a Ubuntu 22.04 machine. -1. Because, in this example, we are running the agent on the same server as the Postgres instance we're monitoring, it's faster and more secure to have the EDB Postgres AI agent use Postgres local auth rather than set up password auth over TCP/IP. +1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have the EDB Postgres AI agent use Postgres local auth rather than set up password auth over TCP/IP. !!! Note In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production you'd want to use a minimally privileged user created specifically for the purposes of running the Beacon agent on the server. From 028992320b484ae61a95e9999ad29fb83f868d9d Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Thu, 16 May 2024 14:36:09 -0400 Subject: [PATCH 32/67] Typo. --- advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx index b327c1de9e7..fcb2bb65218 100644 --- a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx @@ -216,7 +216,7 @@ What follows is an example of how to run EDB Postgres AI agent as a service, spe 1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have the EDB Postgres AI agent use Postgres local auth rather than set up password auth over TCP/IP. !!! Note - In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production you'd want to use a minimally privileged user created specifically for the purposes of running the Beacon agent on the server. + In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running the Beacon agent on the server. !!! To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: From 6de626bc5010a5b059711544b3fb04b53ef98c52 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Fri, 17 May 2024 13:27:53 -0400 Subject: [PATCH 33/67] Update terminology. --- advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx index fcb2bb65218..60e9fb832d4 100644 --- a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx @@ -4,7 +4,7 @@ navTitle: Monitoring self-managed deployments deepToC: true --- -To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine agent configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. +To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring the EDB Postgres AI Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. From 1d7324ef2c2b1529da68b7bab04c447bfcb7db99 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Mon, 20 May 2024 10:21:25 -0400 Subject: [PATCH 34/67] move file to relevant folder. --- .../databases/pai-agent-self-managed.mdx | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename advocacy_docs/{postgres-ai => edb-postgres-ai}/databases/pai-agent-self-managed.mdx (100%) diff --git a/advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx similarity index 100% rename from advocacy_docs/postgres-ai/databases/pai-agent-self-managed.mdx rename to advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx From 62a554caba335a6850ea97b13d73ee0c9ab220aa Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 10:36:35 -0400 Subject: [PATCH 35/67] Update advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../edb-postgres-ai/databases/pai-agent-self-managed.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx index 60e9fb832d4..9574218e336 100644 --- a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx @@ -193,7 +193,7 @@ The message in the second to last line of the logs above specifies that we are u 7. Run the Beacon agent. - In the `beacon_agent.yaml` file, replace `"stdout"` in `agent:beacon_server` with `"beacon.biganimal.com:443"`: + In the `beacon_agent.yaml` file, replace `"stdout" with `"beacon.biganimal.com:443"`: ``` agent: From 2e3283c54c03ffcae605d69d7c8eb87507b7933e Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 13:05:25 -0400 Subject: [PATCH 36/67] Integrated sonotley's suggestions. --- .../databases/pai-agent-self-managed.mdx | 51 ++++++++++--------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx index 9574218e336..13b4a159b12 100644 --- a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx @@ -1,12 +1,12 @@ --- -title: Installing EDB Postgres AI Agent for monitoring self-managed Postgres deployments +title: Installing Beacon Agent for monitoring self-managed Postgres deployments navTitle: Monitoring self-managed deployments deepToC: true --- To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. -This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring the EDB Postgres AI Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. +This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. ## Setting up a machine user for estate ingestion in EDB Postgres AI Console @@ -24,28 +24,28 @@ This guide walks through the procedure for setting up a machine user in the EDB 7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. -8. Your machine user is ready for estate ingestions from the EDB Postgres AI Agent. +8. Your machine user is ready for estate ingestions from Beacon Agent. -## Installing, configuring, testing, and running the EDB Postgres AI agent +## Installing, configuring, testing, and running Beacon Agent -The following steps walk you through how to install and configure the EDB Postgres AI agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. +The following steps walk you through how to install and configure Beacon Agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. 1. Establish a connection between your system and the Cloudsmith repository for EDB Postgres AI. First, find your EnterpriseDB Repos 2.0 token [here](https://www.enterprisedb.com/repos-downloads). - Next, download and run a script to configure your system to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token: + Next, download and run a script to configure your system to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token and replace `` with your subscription type (`standard` or `enterprise`): For RHEL-like or SLES: ``` - curl -1sLf 'https://downloads.enterprisedb.com//dev/setup.rpm.sh' | sudo -E bash + curl -1sLf 'https://downloads.enterprisedb.com///setup.rpm.sh' | sudo -E bash ``` For Debian or Ubuntu: ``` - curl -1sLf 'https://downloads.enterprisedb.com//dev/setup.deb.sh' | sudo -E bash + curl -1sLf 'https://downloads.enterprisedb.com//>/setup.deb.sh' | sudo -E bash ``` 2. Install the `beacon-agent` package: @@ -74,15 +74,15 @@ The following steps walk you through how to install and configure the EDB Postgr apt install beacon-agent ``` -3. Configure EDB Postgres AI Agent and database connections. +3. Configure Beacon Agent and database connections. - First, create a beacon config in your home directory: + First, create a Beacon config in your home directory: ``` mkdir ${HOME}/.beacon ``` - Next, configure the beacon agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: + Next, configure Beacon Agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: ``` export BEACON_AGENT_ACCESS_KEY= @@ -90,9 +90,9 @@ The following steps walk you through how to install and configure the EDB Postgr beacon-agent setup -file="$HOME/.beacon/beacon_agent.yaml" ``` - These commands set up the necessary parameters for the Beacon agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. + These commands set up the necessary parameters for Beacon Agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. - During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for the EDB Postgres AI Agent to communicate with EDB Postgres AI and register successfully. + During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for Beacon Agent to communicate with the Beacon server and register with the project successfully. If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your EDB Postgres AI project. @@ -121,7 +121,7 @@ The following steps walk you through how to install and configure the EDB Postgr - "" ``` - Here is an example with a database named `john_laptop_0`: + Here is an example with a database named `sales_reporting`: ``` agent: access_key: $BEACON_AGENT_ACCESS_KEY @@ -135,7 +135,7 @@ agent: provider: onprem: databases: - john_laptop_0: + sales_reporting: dsn: “$DSN” tags: - "foo" @@ -146,9 +146,9 @@ provider: poll_interval: 5m0s ``` -6. Test the EDB Postgres AI Agent locally. +6. Test Beacon Agent locally. - As an initial smoke test of the agent, it can send the ingestions that it would send back to the beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. + As an initial smoke test of the agent, it can send the ingestions that it would send back to the Beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. You can run the agent in stdout mode by modifying the `beacon_agent.yaml` file generated previously to have an `agent.beacon_server` value of `"stdout"`. A truncated example of this would be: @@ -163,7 +163,7 @@ provider: beacon-agent ``` - You should see output similar to the following: + You should see output similar to the following eventually (it can take around 5 minutes to see the last few lines): ``` {"level":"debug","data":"$BEACON_AGENT_ACCESS_KEY","time":"2024-05-08T18:40:34Z","message":"expanding environment variable in configuration"} @@ -191,7 +191,7 @@ provider: The message in the second to last line of the logs above specifies that we are using the log client and not actually sending ingestions. !!! -7. Run the Beacon agent. +7. Run Beacon Agent. In the `beacon_agent.yaml` file, replace `"stdout" with `"beacon.biganimal.com:443"`: @@ -205,18 +205,18 @@ The message in the second to last line of the logs above specifies that we are u ## Running Postgres AI agent as a service -To have EDB Postgres AI agent run automatically on startup and restart after error, you need to make it a service. +To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. !!! Note Future versions of the agent package may set this up automatically. !!! -What follows is an example of how to run EDB Postgres AI agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running the Beacon agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed EDB Postgres AI Agent on a Ubuntu 22.04 machine. +What follows is an example of how to run Beacon Agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running Beacon Agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed Beacon Agent on a Ubuntu 22.04 machine. -1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have the EDB Postgres AI agent use Postgres local auth rather than set up password auth over TCP/IP. +1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have Beacon agent use Postgres local auth rather than set up password auth over TCP/IP. !!! Note - In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running the Beacon agent on the server. + In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running Beacon Agent on the server. !!! To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: @@ -235,7 +235,7 @@ What follows is an example of how to run EDB Postgres AI agent as a service, spe This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). - Now Postgres is configured to allow local authentication so that the Beacon agent can access it as a service. + Now Postgres is configured to allow local authentication so that Beacon Agent can access it as a service. 2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: @@ -268,7 +268,8 @@ User=ubuntu WorkingDirectory=/home/ubuntu/ -Environment='BEACON_AGENT_ACCESS_KEY=' +Environment='BEACON_AGENT_ACCESS_KEY=' +Environment='DSN=' # Command to run the application From 918bf1f2d67c3fdc38003d6c7c0db92b212eb3a2 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 16:11:13 -0400 Subject: [PATCH 37/67] Created agent folder, moved file to new folder, deleted old folder and agent file. --- .../edb-postgres-ai/console/agent/index.mdx | 10 + .../console/agent/pai-agent-self-managed.mdx | 305 ++++++++++++++++++ 2 files changed, 315 insertions(+) create mode 100644 advocacy_docs/edb-postgres-ai/console/agent/index.mdx create mode 100644 advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx diff --git a/advocacy_docs/edb-postgres-ai/console/agent/index.mdx b/advocacy_docs/edb-postgres-ai/console/agent/index.mdx new file mode 100644 index 00000000000..3a964e51fe7 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/agent/index.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI Console - Agent +navTitle: Agent +description: The EDB Postgres AI Console Agent allows you to expand your EDB Postgres AI Console's visibility to your on-premises and in-cloud Postgres databases. +--- + +## What is the EDB Postgres AI Agent + +_The EDB Postgres AI Agent is a component of the EDB Postgres AI Console. The Agent is installed alongside your existing On-prem or in-cloud Postgres databases and makes them observable on the EDB Postgres AI Console's single pane of glass view._ + diff --git a/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx new file mode 100644 index 00000000000..13b4a159b12 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx @@ -0,0 +1,305 @@ +--- +title: Installing Beacon Agent for monitoring self-managed Postgres deployments +navTitle: Monitoring self-managed deployments +deepToC: true +--- + +To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. + +This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. + +## Setting up a machine user for estate ingestion in EDB Postgres AI Console + +1. In the EDB Postgres AI Console, using your avatar's dropdown, select the **User Management** page. + +2. Select the **Add New User** button. + +3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally select the **Add User** button. + +4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. + +5. Next, go to the project from which you want to monitor the database, select the **Users** tab, and locate the machine user you just created. + +6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. + +7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. + +8. Your machine user is ready for estate ingestions from Beacon Agent. + +## Installing, configuring, testing, and running Beacon Agent + +The following steps walk you through how to install and configure Beacon Agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. + +1. Establish a connection between your system and the Cloudsmith repository for EDB Postgres AI. + + First, find your EnterpriseDB Repos 2.0 token [here](https://www.enterprisedb.com/repos-downloads). + + Next, download and run a script to configure your system to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token and replace `` with your subscription type (`standard` or `enterprise`): + + For RHEL-like or SLES: + + ``` + curl -1sLf 'https://downloads.enterprisedb.com///setup.rpm.sh' | sudo -E bash + ``` + + For Debian or Ubuntu: + + ``` + curl -1sLf 'https://downloads.enterprisedb.com//>/setup.deb.sh' | sudo -E bash + ``` + +2. Install the `beacon-agent` package: + + For RHEL-like: + + ``` + dnf install beacon-agent + ``` + + Or if dnf isn't available: + + ``` + yum install beacon-agent + ``` + + For SLES: + + ``` + zypper install beacon-agent + ``` + + For Debian or Ubuntu: + + ``` + apt install beacon-agent + ``` + +3. Configure Beacon Agent and database connections. + + First, create a Beacon config in your home directory: + + ``` + mkdir ${HOME}/.beacon + ``` + + Next, configure Beacon Agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: + + ``` + export BEACON_AGENT_ACCESS_KEY= + export BEACON_AGENT_PROJECT_ID= + beacon-agent setup -file="$HOME/.beacon/beacon_agent.yaml" + ``` + + These commands set up the necessary parameters for Beacon Agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. + + During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for Beacon Agent to communicate with the Beacon server and register with the project successfully. + + If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your EDB Postgres AI project. + + +4. For security best practices, create a `DSN` environmental variable for each database (`DSN1`, `DSN2`, and so on) you want to monitor and set each to the correct corresponding dsn. + + ``` + export DSN1= + ``` + +5. Open your `$HOME/.beacon/beacon_agent.yaml` configuration file and add an entry for each database you want to connect to, including the environmental variable for each database's dsn. + + Entries under `databases` utilize the following format: + + ``` + databases: + : + dsn: "$DSN1" + tags: + - "" + - "" + : + dsn: "$DSN2" + tags: + - "" + - "" + ``` + + Here is an example with a database named `sales_reporting`: +``` +agent: + access_key: $BEACON_AGENT_ACCESS_KEY + access_key_grpc_header: "x-access-key" + batch_size: 100 + beacon_server: "beacon.biganimal.com:443" + feature_flag_interval: 10m0s + project_id: "" + providers: + - "onprem" +provider: + onprem: + databases: + sales_reporting: + dsn: “$DSN” + tags: + - "foo" + - "bar" + host: + resource_id: "Johns-MBP.lan" + tags: [] + poll_interval: 5m0s +``` + +6. Test Beacon Agent locally. + + As an initial smoke test of the agent, it can send the ingestions that it would send back to the Beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. + + You can run the agent in stdout mode by modifying the `beacon_agent.yaml` file generated previously to have an `agent.beacon_server` value of `"stdout"`. A truncated example of this would be: + + ``` + agent: + beacon_server: "stdout" + ``` + + Next, run the agent in this mode using the following command: + + ``` + beacon-agent + ``` + + You should see output similar to the following eventually (it can take around 5 minutes to see the last few lines): + +``` +{"level":"debug","data":"$BEACON_AGENT_ACCESS_KEY","time":"2024-05-08T18:40:34Z","message":"expanding environment variable in configuration"} +{"level":"info","path":"/healthz","time":1715193634,"msg":"serving liveness probe"} +{"level":"info","path":"/readyz","time":1715193634,"msg":"serving readiness probe"} +{"level":"info","version":"v1.51.0-snapshot8986075626.97.1.166215e","time":1715193634,"msg":"starting beacon agent"} +{"level":"info","spiffe_enabled":false,"time":1715193634,"msg":"configuring tls"} +{"level":"info","server":"stdout","time":1715193634,"msg":"connecting to beacon service"} +{"level":"info","address":":8081","time":1715193634,"msg":"starting probe server"} +{"level":"info","target":"stdout","time":1715193634,"msg":"connected to beacon server"} +{"level":"info","time":1715193634,"msg":"verifying connection to beacon server"} +{"level":"info","project":"echo","time":1715193634,"msg":"verified connection to beacon server"} +{"level":"info","interval":"10m0s","time":1715193634,"msg":"loading feature flags periodically"} +{"level":"info","time":1715193634,"msg":"fetching feature flags"} +{"level":"info","feature_flags":{"echo_flag":"test","second_flag":false},"time":1715193634,"msg":"loaded feature flags"} +{"level":"info","id":"onprem","time":1715193634,"msg":"starting provider"} +{"level":"info","disable_partitioning":false,"batch_size":100,"interval":"10s","time":1715193634,"msg":"starting batch exporter"} +{"level":"info","provider_id":"onprem","time":1715193634,"msg":"registering ingestion worker in pool"} +{"level":"info","provider":"onprem","time":1715193634,"msg":"starting provider worker"} +{"level":"info","ingestions":[{"version":"v0.1.0","type":"onprem/host","id":"ip-10-0-128-121","metadata":{"Data":{"OnPremHostMetadata":{"hostname":"ip-10-0-128-121","operating_system":"linux","platform":"ubuntu","platform_family":"debian","platform_version":"22.04","cpu_limit":1}}}}],"time":1715193934,"msg":"sending ingestions via log client (not actually sending)"} +{"level":"info","successful_ingestions":1,"failed_ingestions":0,"time":1715193934,"msg":"exported ingestions"} +``` + +!!! Note +The message in the second to last line of the logs above specifies that we are using the log client and not actually sending ingestions. +!!! + +7. Run Beacon Agent. + + In the `beacon_agent.yaml` file, replace `"stdout" with `"beacon.biganimal.com:443"`: + + ``` + agent: + beacon_server: "beacon.biganimal.com:443" + + ``` + +8. Follow the logs to monitor the ingestions. + +## Running Postgres AI agent as a service + +To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. + +!!! Note +Future versions of the agent package may set this up automatically. +!!! + +What follows is an example of how to run Beacon Agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running Beacon Agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed Beacon Agent on a Ubuntu 22.04 machine. + +1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have Beacon agent use Postgres local auth rather than set up password auth over TCP/IP. + + !!! Note + In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running Beacon Agent on the server. + !!! + + To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: + + ``` + sudo su postgres + psql -c 'CREATE USER ubuntu' + exit + ``` + + To complete the setup for local authentication with Postgres, you need to ensure your `pg_hba.conf` file is configured to allow Unix-domain socket connections. Please verify or update the following line in your `pg_hba.conf` file: + + ``` + local all all peer + ``` + + This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). + + Now Postgres is configured to allow local authentication so that Beacon Agent can access it as a service. + +2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: + +``` +sudo vi /etc/systemd/beacon-agent.service +``` + +3. Populate the new file as follows, specifying the BEACON_AGENT_ACCESS_KEY, then save and exit the editor: + +``` +[Unit] + +Description=The Postgres AI Agent + +# After networking because we need that + +After=network.target + +[Service] + +# Simple services don't do any forking / background nonsense + +Type=simple + +# User with which to run the service + +User=ubuntu + +# Set the working directory for the application + +WorkingDirectory=/home/ubuntu/ + +Environment='BEACON_AGENT_ACCESS_KEY=' +Environment='DSN=' + +# Command to run the application + +ExecStart=/usr/local/bin/beacon-agent + +# Restart policy, only on failure + +Restart=on-failure + +RestartSec=60 + +[Install] + +# Start the service before we get to multi-user mode + +WantedBy=multi-user.target +``` + +4. Run the following commands to reload, enable, and start your new service: + +``` +sudo systemctl daemon-reload +sudo systemctl enable beacon-agent.service +sudo systemctl start beacon-agent.service +``` + +5. Your agent should now be running as a service. Check on the logs by running the following command: + +``` +journalctl -u beacon-agent.service +``` + + From 4660bd5590f65f955a65fa0221d91c121bf2c082 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 16:15:33 -0400 Subject: [PATCH 38/67] Update advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx Co-authored-by: Simon Notley <43099400+sonotley@users.noreply.github.com> --- .../edb-postgres-ai/databases/pai-agent-self-managed.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx index 13b4a159b12..0e87ecd4808 100644 --- a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx @@ -45,7 +45,7 @@ The following steps walk you through how to install and configure Beacon Agent, For Debian or Ubuntu: ``` - curl -1sLf 'https://downloads.enterprisedb.com//>/setup.deb.sh' | sudo -E bash + curl -1sLf 'https://downloads.enterprisedb.com///setup.deb.sh' | sudo -E bash ``` 2. Install the `beacon-agent` package: From 00ed722f30ae53d23c74e3a74beaa0370edc0811 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 16:30:39 -0400 Subject: [PATCH 39/67] Fixed some Beacon refs. --- .../console/agent/pai-agent-self-managed.mdx | 21 +++++-------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx index 13b4a159b12..660f08d7571 100644 --- a/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx +++ b/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx @@ -4,9 +4,9 @@ navTitle: Monitoring self-managed deployments deepToC: true --- -To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. +To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and Beacon Agent configured to access the database and report back to the EDB Postgres AI Console. -This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. +This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running Beacon Agent as a service. ## Setting up a machine user for estate ingestion in EDB Postgres AI Console @@ -14,7 +14,7 @@ This guide walks through the procedure for setting up a machine user in the EDB 2. Select the **Add New User** button. -3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally select the **Add User** button. +3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally, select the **Add User** button. 4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. @@ -203,7 +203,7 @@ The message in the second to last line of the logs above specifies that we are u 8. Follow the logs to monitor the ingestions. -## Running Postgres AI agent as a service +## Running Beacon Agent as a service To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. @@ -247,44 +247,33 @@ sudo vi /etc/systemd/beacon-agent.service ``` [Unit] - -Description=The Postgres AI Agent +Description=Beacon Agent # After networking because we need that - After=network.target [Service] - # Simple services don't do any forking / background nonsense - Type=simple # User with which to run the service - User=ubuntu # Set the working directory for the application - WorkingDirectory=/home/ubuntu/ Environment='BEACON_AGENT_ACCESS_KEY=' Environment='DSN=' # Command to run the application - ExecStart=/usr/local/bin/beacon-agent # Restart policy, only on failure - Restart=on-failure - RestartSec=60 [Install] - # Start the service before we get to multi-user mode - WantedBy=multi-user.target ``` From 3361a81b9beef037ca63d467f68124728cc4c473 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 16:57:05 -0400 Subject: [PATCH 40/67] fix deletion --- .../edb-postgres-ai/console/agent.mdx | 10 - .../databases/pai-agent-self-managed.mdx | 305 ------------------ 2 files changed, 315 deletions(-) delete mode 100644 advocacy_docs/edb-postgres-ai/console/agent.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx diff --git a/advocacy_docs/edb-postgres-ai/console/agent.mdx b/advocacy_docs/edb-postgres-ai/console/agent.mdx deleted file mode 100644 index 3a964e51fe7..00000000000 --- a/advocacy_docs/edb-postgres-ai/console/agent.mdx +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: EDB Postgres AI Console - Agent -navTitle: Agent -description: The EDB Postgres AI Console Agent allows you to expand your EDB Postgres AI Console's visibility to your on-premises and in-cloud Postgres databases. ---- - -## What is the EDB Postgres AI Agent - -_The EDB Postgres AI Agent is a component of the EDB Postgres AI Console. The Agent is installed alongside your existing On-prem or in-cloud Postgres databases and makes them observable on the EDB Postgres AI Console's single pane of glass view._ - diff --git a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx deleted file mode 100644 index 0e87ecd4808..00000000000 --- a/advocacy_docs/edb-postgres-ai/databases/pai-agent-self-managed.mdx +++ /dev/null @@ -1,305 +0,0 @@ ---- -title: Installing Beacon Agent for monitoring self-managed Postgres deployments -navTitle: Monitoring self-managed deployments -deepToC: true ---- - -To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and an EDB Postgres AI agent configured to access the database and report back to the EDB Postgres AI Console. - -This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running the EDB Postgres AI Agent as a service. - -## Setting up a machine user for estate ingestion in EDB Postgres AI Console - -1. In the EDB Postgres AI Console, using your avatar's dropdown, select the **User Management** page. - -2. Select the **Add New User** button. - -3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally select the **Add User** button. - -4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. - -5. Next, go to the project from which you want to monitor the database, select the **Users** tab, and locate the machine user you just created. - -6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. - -7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. - -8. Your machine user is ready for estate ingestions from Beacon Agent. - -## Installing, configuring, testing, and running Beacon Agent - -The following steps walk you through how to install and configure Beacon Agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. - -1. Establish a connection between your system and the Cloudsmith repository for EDB Postgres AI. - - First, find your EnterpriseDB Repos 2.0 token [here](https://www.enterprisedb.com/repos-downloads). - - Next, download and run a script to configure your system to access the `beacon-agent` package from the repository. Be sure to replace `` with your EDB Repos 2.0 token and replace `` with your subscription type (`standard` or `enterprise`): - - For RHEL-like or SLES: - - ``` - curl -1sLf 'https://downloads.enterprisedb.com///setup.rpm.sh' | sudo -E bash - ``` - - For Debian or Ubuntu: - - ``` - curl -1sLf 'https://downloads.enterprisedb.com///setup.deb.sh' | sudo -E bash - ``` - -2. Install the `beacon-agent` package: - - For RHEL-like: - - ``` - dnf install beacon-agent - ``` - - Or if dnf isn't available: - - ``` - yum install beacon-agent - ``` - - For SLES: - - ``` - zypper install beacon-agent - ``` - - For Debian or Ubuntu: - - ``` - apt install beacon-agent - ``` - -3. Configure Beacon Agent and database connections. - - First, create a Beacon config in your home directory: - - ``` - mkdir ${HOME}/.beacon - ``` - - Next, configure Beacon Agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: - - ``` - export BEACON_AGENT_ACCESS_KEY= - export BEACON_AGENT_PROJECT_ID= - beacon-agent setup -file="$HOME/.beacon/beacon_agent.yaml" - ``` - - These commands set up the necessary parameters for Beacon Agent and execute the setup command to generate the configuration file(`beacon_agent.yaml) in the specified directory. - - During this setup process, an authentication attempt occurs, utilizing the provided access key and project ID. This authentication is necessary for Beacon Agent to communicate with the Beacon server and register with the project successfully. - - If the commands execute as expected, you should see a message indicating that you have authenticated successfully to your EDB Postgres AI project. - - -4. For security best practices, create a `DSN` environmental variable for each database (`DSN1`, `DSN2`, and so on) you want to monitor and set each to the correct corresponding dsn. - - ``` - export DSN1= - ``` - -5. Open your `$HOME/.beacon/beacon_agent.yaml` configuration file and add an entry for each database you want to connect to, including the environmental variable for each database's dsn. - - Entries under `databases` utilize the following format: - - ``` - databases: - : - dsn: "$DSN1" - tags: - - "" - - "" - : - dsn: "$DSN2" - tags: - - "" - - "" - ``` - - Here is an example with a database named `sales_reporting`: -``` -agent: - access_key: $BEACON_AGENT_ACCESS_KEY - access_key_grpc_header: "x-access-key" - batch_size: 100 - beacon_server: "beacon.biganimal.com:443" - feature_flag_interval: 10m0s - project_id: "" - providers: - - "onprem" -provider: - onprem: - databases: - sales_reporting: - dsn: “$DSN” - tags: - - "foo" - - "bar" - host: - resource_id: "Johns-MBP.lan" - tags: [] - poll_interval: 5m0s -``` - -6. Test Beacon Agent locally. - - As an initial smoke test of the agent, it can send the ingestions that it would send back to the Beacon server to stdout instead. This allows you to quickly confirm if the agent is successfully able to gather ingestions and what those ingestions look like. - - You can run the agent in stdout mode by modifying the `beacon_agent.yaml` file generated previously to have an `agent.beacon_server` value of `"stdout"`. A truncated example of this would be: - - ``` - agent: - beacon_server: "stdout" - ``` - - Next, run the agent in this mode using the following command: - - ``` - beacon-agent - ``` - - You should see output similar to the following eventually (it can take around 5 minutes to see the last few lines): - -``` -{"level":"debug","data":"$BEACON_AGENT_ACCESS_KEY","time":"2024-05-08T18:40:34Z","message":"expanding environment variable in configuration"} -{"level":"info","path":"/healthz","time":1715193634,"msg":"serving liveness probe"} -{"level":"info","path":"/readyz","time":1715193634,"msg":"serving readiness probe"} -{"level":"info","version":"v1.51.0-snapshot8986075626.97.1.166215e","time":1715193634,"msg":"starting beacon agent"} -{"level":"info","spiffe_enabled":false,"time":1715193634,"msg":"configuring tls"} -{"level":"info","server":"stdout","time":1715193634,"msg":"connecting to beacon service"} -{"level":"info","address":":8081","time":1715193634,"msg":"starting probe server"} -{"level":"info","target":"stdout","time":1715193634,"msg":"connected to beacon server"} -{"level":"info","time":1715193634,"msg":"verifying connection to beacon server"} -{"level":"info","project":"echo","time":1715193634,"msg":"verified connection to beacon server"} -{"level":"info","interval":"10m0s","time":1715193634,"msg":"loading feature flags periodically"} -{"level":"info","time":1715193634,"msg":"fetching feature flags"} -{"level":"info","feature_flags":{"echo_flag":"test","second_flag":false},"time":1715193634,"msg":"loaded feature flags"} -{"level":"info","id":"onprem","time":1715193634,"msg":"starting provider"} -{"level":"info","disable_partitioning":false,"batch_size":100,"interval":"10s","time":1715193634,"msg":"starting batch exporter"} -{"level":"info","provider_id":"onprem","time":1715193634,"msg":"registering ingestion worker in pool"} -{"level":"info","provider":"onprem","time":1715193634,"msg":"starting provider worker"} -{"level":"info","ingestions":[{"version":"v0.1.0","type":"onprem/host","id":"ip-10-0-128-121","metadata":{"Data":{"OnPremHostMetadata":{"hostname":"ip-10-0-128-121","operating_system":"linux","platform":"ubuntu","platform_family":"debian","platform_version":"22.04","cpu_limit":1}}}}],"time":1715193934,"msg":"sending ingestions via log client (not actually sending)"} -{"level":"info","successful_ingestions":1,"failed_ingestions":0,"time":1715193934,"msg":"exported ingestions"} -``` - -!!! Note -The message in the second to last line of the logs above specifies that we are using the log client and not actually sending ingestions. -!!! - -7. Run Beacon Agent. - - In the `beacon_agent.yaml` file, replace `"stdout" with `"beacon.biganimal.com:443"`: - - ``` - agent: - beacon_server: "beacon.biganimal.com:443" - - ``` - -8. Follow the logs to monitor the ingestions. - -## Running Postgres AI agent as a service - -To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. - -!!! Note -Future versions of the agent package may set this up automatically. -!!! - -What follows is an example of how to run Beacon Agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running Beacon Agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed Beacon Agent on a Ubuntu 22.04 machine. - -1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have Beacon agent use Postgres local auth rather than set up password auth over TCP/IP. - - !!! Note - In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running Beacon Agent on the server. - !!! - - To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: - - ``` - sudo su postgres - psql -c 'CREATE USER ubuntu' - exit - ``` - - To complete the setup for local authentication with Postgres, you need to ensure your `pg_hba.conf` file is configured to allow Unix-domain socket connections. Please verify or update the following line in your `pg_hba.conf` file: - - ``` - local all all peer - ``` - - This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). - - Now Postgres is configured to allow local authentication so that Beacon Agent can access it as a service. - -2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: - -``` -sudo vi /etc/systemd/beacon-agent.service -``` - -3. Populate the new file as follows, specifying the BEACON_AGENT_ACCESS_KEY, then save and exit the editor: - -``` -[Unit] - -Description=The Postgres AI Agent - -# After networking because we need that - -After=network.target - -[Service] - -# Simple services don't do any forking / background nonsense - -Type=simple - -# User with which to run the service - -User=ubuntu - -# Set the working directory for the application - -WorkingDirectory=/home/ubuntu/ - -Environment='BEACON_AGENT_ACCESS_KEY=' -Environment='DSN=' - -# Command to run the application - -ExecStart=/usr/local/bin/beacon-agent - -# Restart policy, only on failure - -Restart=on-failure - -RestartSec=60 - -[Install] - -# Start the service before we get to multi-user mode - -WantedBy=multi-user.target -``` - -4. Run the following commands to reload, enable, and start your new service: - -``` -sudo systemctl daemon-reload -sudo systemctl enable beacon-agent.service -sudo systemctl start beacon-agent.service -``` - -5. Your agent should now be running as a service. Check on the logs by running the following command: - -``` -journalctl -u beacon-agent.service -``` - - From 7c62caa81aac4f94aeb1fca2445d870ad0ec66e3 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 22 May 2024 10:16:00 +0100 Subject: [PATCH 41/67] Reorg to a Agent section Signed-off-by: Dj Walker-Morgan --- .../console/agent/agent-as-a-service.mdx | 94 +++++++++++++ .../console/agent/create-machine-user.mdx | 21 +++ .../edb-postgres-ai/console/agent/index.mdx | 13 +- ...ent-self-managed.mdx => install-agent.mdx} | 124 +----------------- 4 files changed, 132 insertions(+), 120 deletions(-) create mode 100644 advocacy_docs/edb-postgres-ai/console/agent/agent-as-a-service.mdx create mode 100644 advocacy_docs/edb-postgres-ai/console/agent/create-machine-user.mdx rename advocacy_docs/edb-postgres-ai/console/agent/{pai-agent-self-managed.mdx => install-agent.mdx} (56%) diff --git a/advocacy_docs/edb-postgres-ai/console/agent/agent-as-a-service.mdx b/advocacy_docs/edb-postgres-ai/console/agent/agent-as-a-service.mdx new file mode 100644 index 00000000000..cec65fe6327 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/agent/agent-as-a-service.mdx @@ -0,0 +1,94 @@ +--- +title: Running Beacon Agent as a service +description: How to run Beacon Agent as a service on Ubuntu 22.04 +--- + +## Running Beacon Agent as a service + +To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. + +!!! Note +Future versions of the agent package may set this up automatically. +!!! + +What follows is an example of how to run Beacon Agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running Beacon Agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed Beacon Agent on a Ubuntu 22.04 machine. + +1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have Beacon agent use Postgres local auth rather than set up password auth over TCP/IP. + + !!! Note + In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running Beacon Agent on the server. + !!! + + To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: + + ``` + sudo su postgres + psql -c 'CREATE USER ubuntu' + exit + ``` + + To complete the setup for local authentication with Postgres, you need to ensure your `pg_hba.conf` file is configured to allow Unix-domain socket connections. Please verify or update the following line in your `pg_hba.conf` file: + + ``` + local all all peer + ``` + + This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). + + Now Postgres is configured to allow local authentication so that Beacon Agent can access it as a service. + +2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: + +``` +sudo vi /etc/systemd/beacon-agent.service +``` + +3. Populate the new file as follows, specifying the BEACON_AGENT_ACCESS_KEY, then save and exit the editor: + +``` +[Unit] +Description=Beacon Agent + +# After networking because we need that +After=network.target + +[Service] +# Simple services don't do any forking / background nonsense +Type=simple + +# User with which to run the service +User=ubuntu + +# Set the working directory for the application +WorkingDirectory=/home/ubuntu/ + +Environment='BEACON_AGENT_ACCESS_KEY=' +Environment='DSN=' + +# Command to run the application +ExecStart=/usr/local/bin/beacon-agent + +# Restart policy, only on failure +Restart=on-failure +RestartSec=60 + +[Install] +# Start the service before we get to multi-user mode +WantedBy=multi-user.target +``` + +4. Run the following commands to reload, enable, and start your new service: + +``` +sudo systemctl daemon-reload +sudo systemctl enable beacon-agent.service +sudo systemctl start beacon-agent.service +``` + +5. Your agent should now be running as a service. Check on the logs by running the following command: + +``` +journalctl -u beacon-agent.service +``` + + diff --git a/advocacy_docs/edb-postgres-ai/console/agent/create-machine-user.mdx b/advocacy_docs/edb-postgres-ai/console/agent/create-machine-user.mdx new file mode 100644 index 00000000000..56d05701536 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/agent/create-machine-user.mdx @@ -0,0 +1,21 @@ +--- +title: Creating a machine user +description: Learn how to create a machine user in EDB Postgres AI Console to enable Beacon Agent. +--- + + +1. In the EDB Postgres AI Console, using your avatar's dropdown, select the **User Management** page. + +2. Select the **Add New User** button. + +3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally, select the **Add User** button. + +4. After you get your access key, store it somewhere securely, as EDB does only lets you view access keys at creation time. If you lose your record of an access key, you will have to get a new/replacement key by regenerating it. + +5. Next, go to the project from which you want to monitor the database, select the **Users** tab, and locate the machine user you just created. + +6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. + +7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. + +8. Your machine user is ready for estate ingestions from Beacon Agent. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/console/agent/index.mdx b/advocacy_docs/edb-postgres-ai/console/agent/index.mdx index 3a964e51fe7..c3f666da77c 100644 --- a/advocacy_docs/edb-postgres-ai/console/agent/index.mdx +++ b/advocacy_docs/edb-postgres-ai/console/agent/index.mdx @@ -1,10 +1,17 @@ --- title: EDB Postgres AI Console - Agent navTitle: Agent -description: The EDB Postgres AI Console Agent allows you to expand your EDB Postgres AI Console's visibility to your on-premises and in-cloud Postgres databases. +description: Installing, configuring, testing, and running Beacon Agent +indexCards: simple +navigation: +- create-machine-user +- install-agent +- agent-as-a-service --- -## What is the EDB Postgres AI Agent +To monitor your self-managed Postgres database with Beacon Agent, you will need to: -_The EDB Postgres AI Agent is a component of the EDB Postgres AI Console. The Agent is installed alongside your existing On-prem or in-cloud Postgres databases and makes them observable on the EDB Postgres AI Console's single pane of glass view._ +* [Create a machine user](create-machine-user) in the EDB Postgres AI Console. This will provide an access key for the agent. +* [Install Beacon Agent](install-agent) on the server where your Postgres instance is running. You will use the access key to enable the agent to communicate with the EDB Postgres AI Estate service. +* [Run Beacon Agent as a service](agent-as-a-service) to have it start automatically on system startup and restart after errors. diff --git a/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx b/advocacy_docs/edb-postgres-ai/console/agent/install-agent.mdx similarity index 56% rename from advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx rename to advocacy_docs/edb-postgres-ai/console/agent/install-agent.mdx index 660f08d7571..9dc69e0dada 100644 --- a/advocacy_docs/edb-postgres-ai/console/agent/pai-agent-self-managed.mdx +++ b/advocacy_docs/edb-postgres-ai/console/agent/install-agent.mdx @@ -1,34 +1,14 @@ --- -title: Installing Beacon Agent for monitoring self-managed Postgres deployments -navTitle: Monitoring self-managed deployments -deepToC: true +title: Installing Beacon Agent +description: Installing, configuring, testing, and running Beacon Agent --- -To monitor a self-managed Postgres instance using EDB Postgres AI, you need a machine user configured correctly in the EDB Postgres AI Console and Beacon Agent configured to access the database and report back to the EDB Postgres AI Console. +The following steps walk you through how to install and configure Beacon Agent, test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. -This guide walks through the procedure for setting up a machine user in the EDB Postgres AI Console before configuring Beacon Agent. There are also optional instructions for running Beacon Agent as a service. +Before you begin, you need to have the following: -## Setting up a machine user for estate ingestion in EDB Postgres AI Console - -1. In the EDB Postgres AI Console, using your avatar's dropdown, select the **User Management** page. - -2. Select the **Add New User** button. - -3. On the **Add New User** page, select *Machine User*, give the user a name, check the **Create Access Key** checkbox, give the key a name, and finally, select the **Add User** button. - -4. After you get your access key, store it somewhere securely, as EDB does not persist access keys. You must regenerate the key to get a new one. - -5. Next, go to the project from which you want to monitor the database, select the **Users** tab, and locate the machine user you just created. - -6. Select the edit button to the right of the user entry, then select the **Assign Roles** button. - -7. In the **Assign Project Roles** modal, select the "estate ingester" option and then select the **Submit** button. - -8. Your machine user is ready for estate ingestions from Beacon Agent. - -## Installing, configuring, testing, and running Beacon Agent - -The following steps walk you through how to install and configure Beacon Agent, smoke test locally, and then run the agent to see the results in your Estates page in the EDB Postgres AI Console. +* The access key for a machine user with the `estate ingester` role assigned to it. For more information, see [Creating a machine user](create-machine-user). +* The project ID for the project you want to monitor. You can find this in the URL when you are in the project in the EDB Postgres AI Console. 1. Establish a connection between your system and the Cloudsmith repository for EDB Postgres AI. @@ -82,7 +62,7 @@ The following steps walk you through how to install and configure Beacon Agent, mkdir ${HOME}/.beacon ``` - Next, configure Beacon Agent by setting the access key (the one you stored from the [last section](#setting-up-a-machine-user-for-estate-ingestion-in-edb-postgres-ai-console)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: + Next, configure Beacon Agent by setting the access key (the one you obtained the [Creating a machine user]](create_machine_user)) and project ID, and specify the Beacon config directory for storing the configuration file created. Use the following commands: ``` export BEACON_AGENT_ACCESS_KEY= @@ -202,93 +182,3 @@ The message in the second to last line of the logs above specifies that we are u ``` 8. Follow the logs to monitor the ingestions. - -## Running Beacon Agent as a service - -To have Beacon Agent run automatically on startup and restart after error, you need to make it a service. - -!!! Note -Future versions of the agent package may set this up automatically. -!!! - -What follows is an example of how to run Beacon Agent as a service, specifically on an Ubuntu 22.04 machine. Follow the instructions above for setting up a machine user and then installing, configuring, testing, and running Beacon Agent before moving on to set up Beacon as a service. These instructions assume you have completed the last two sections and installed Beacon Agent on a Ubuntu 22.04 machine. - -1. We are running the agent on the same server as the Postgres instance we're monitoring. So it's faster and more secure to have Beacon agent use Postgres local auth rather than set up password auth over TCP/IP. - - !!! Note - In this example, we use the 'ubuntu' user that is created by default on the EC2 with a default Ubuntu (22.04) machine image. In production, you'd want to use a minimally privileged user explicitly created for the purposes of running Beacon Agent on the server. - !!! - - To configure for local authentication, add the user, `ubuntu`, to Postgres using `psql` and then exit: - - ``` - sudo su postgres - psql -c 'CREATE USER ubuntu' - exit - ``` - - To complete the setup for local authentication with Postgres, you need to ensure your `pg_hba.conf` file is configured to allow Unix-domain socket connections. Please verify or update the following line in your `pg_hba.conf` file: - - ``` - local all all peer - ``` - - This configuration allows any local system user to connect to any database without a password, provided that a corresponding PostgreSQL role with the same name as the system user exists (in this case, user `ubuntu`). - - Now Postgres is configured to allow local authentication so that Beacon Agent can access it as a service. - -2. Create a new file using your text editor of choice (superuser level permissions are necessary here). vi is used in this example: - -``` -sudo vi /etc/systemd/beacon-agent.service -``` - -3. Populate the new file as follows, specifying the BEACON_AGENT_ACCESS_KEY, then save and exit the editor: - -``` -[Unit] -Description=Beacon Agent - -# After networking because we need that -After=network.target - -[Service] -# Simple services don't do any forking / background nonsense -Type=simple - -# User with which to run the service -User=ubuntu - -# Set the working directory for the application -WorkingDirectory=/home/ubuntu/ - -Environment='BEACON_AGENT_ACCESS_KEY=' -Environment='DSN=' - -# Command to run the application -ExecStart=/usr/local/bin/beacon-agent - -# Restart policy, only on failure -Restart=on-failure -RestartSec=60 - -[Install] -# Start the service before we get to multi-user mode -WantedBy=multi-user.target -``` - -4. Run the following commands to reload, enable, and start your new service: - -``` -sudo systemctl daemon-reload -sudo systemctl enable beacon-agent.service -sudo systemctl start beacon-agent.service -``` - -5. Your agent should now be running as a service. Check on the logs by running the following command: - -``` -journalctl -u beacon-agent.service -``` - - From 878d6a96ec1f5fdb3784a82f16bb0ddb0ef0ac64 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 09:31:50 +0100 Subject: [PATCH 42/67] start content for console Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/console/getstarted.mdx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/console/getstarted.mdx b/advocacy_docs/edb-postgres-ai/console/getstarted.mdx index 4c4a2e5c1d1..2c25b5a734c 100644 --- a/advocacy_docs/edb-postgres-ai/console/getstarted.mdx +++ b/advocacy_docs/edb-postgres-ai/console/getstarted.mdx @@ -4,6 +4,10 @@ navTitle: Get Started description: Get started with the EDB Postgres AI Console. --- -Getting Started with the EDB Postgres AI Console. +The EDB Postgres AI Console is a web-based user interface that provides a single pane of glass for managing and monitoring EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases. The EDB Postgres AI Console provides a unified view of the EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases, allowing users to manage and monitor their databases, users, and resources from a single interface. -_The EDB Postgres AI Console is a web-based user interface that provides a single pane of glass for managing and monitoring EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases. The EDB Postgres AI Console provides a unified view of the EDB Postgres AI Database Cloud Service and EDB Postgres AI Databases, allowing users to manage and monitor their databases, users, and resources from a single interface._ \ No newline at end of file +## Accessing the EDB Postgres AI Console + +To access the EDB Postgres AI Console, you will need to have an account with the EDB Postgres AI Database Cloud Service. If you do not have an account, you can sign up for a free trial at [https://www.enterprisedb.com/edb-postgres-ai](https://www.enterprisedb.com/edb-postgres-ai). + +Once you have an account, you can access the EDB Postgres AI Console by navigating to [https://console.enterprisedb.com](https://console.enterprisedb.com) and logging in with your EDB Postgres AI Database Cloud Service credentials. From da4838740746f47347a83aedb87b09eb2335633d Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 15:53:57 -0400 Subject: [PATCH 43/67] Almost complete first draft. Just need options for setup. --- .../console/reference/beacon-agent-ref.mdx | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx diff --git a/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx new file mode 100644 index 00000000000..e78706e1512 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx @@ -0,0 +1,55 @@ +--- +title: beacon-agent +navTitle: beacon-agent +--- + +## `beacon-agent` command + +### Synopsis + +`beacon-agent` runs Beacon Agent or sets up Beacon agent through its subcommand `setup`. + +### Usage + +``` +beacon-agent [command] [options] +``` + +### Commands + +1. **(no subcommand)** - runs Beacon agent. + + - **Usage** + + ``` + beacon-agent + ``` + + - **Description** + + This default mode manages the Beacon Agent process, which sends data ingestions to the Beacon Server and maintains a local log of its activities. + + - **Examples** + + - Start Beacon Agent: + ``` + beacon-agent + ``` +2. **setup** - creates the configuration file and authenticates with the server. + + - **Usage** + + ``` + beacon-agent setup + ``` + + - **Description** + + This command creates the Beacon configuration file and, by default, authenticates the Beacon project with the Beacon server. + + - **Options** + + + + + From 49a2d9910f696c5e139c94937b77184783f748dd Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Tue, 21 May 2024 22:39:23 -0400 Subject: [PATCH 44/67] Added options and examples for setup. --- .../console/reference/beacon-agent-ref.mdx | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx index e78706e1512..4155d293771 100644 --- a/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx +++ b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx @@ -49,7 +49,22 @@ beacon-agent [command] [options] - **Options** + - `-file string=`: Sets the filename of the generated configuration file. (default "beacon_agent.yaml"). + - `-verify=`: Verifies the project's credentials with the Beacon server. + - `--help`: Provides information about optional flags. + + - **Examples** + - Create the configuration file, but don't authenticate the Beacon project with the Beacon Server. + + ``` + beacon-agent setup -verify=false + ``` + + - Create the configuration file with a different name than the default (beacon_agent.yaml). + + ``` + beacon-agent setup -file string='my_beacon_config.yaml' + ``` - From 6c4051aa9ea639070d48a0be83f5436f55e267d5 Mon Sep 17 00:00:00 2001 From: Josh Earlenbaugh Date: Wed, 22 May 2024 11:11:39 -0400 Subject: [PATCH 45/67] Fix -file flag format. --- .../edb-postgres-ai/console/reference/beacon-agent-ref.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx index 4155d293771..d24d44bbcff 100644 --- a/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx +++ b/advocacy_docs/edb-postgres-ai/console/reference/beacon-agent-ref.mdx @@ -63,7 +63,7 @@ beacon-agent [command] [options] - Create the configuration file with a different name than the default (beacon_agent.yaml). ``` - beacon-agent setup -file string='my_beacon_config.yaml' + beacon-agent setup -file my_beacon_config.yaml ``` From 5977d91ca66c768117da3225cfb520fa22036646 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 22 May 2024 14:54:27 +0100 Subject: [PATCH 46/67] First commit of content Signed-off-by: Dj Walker-Morgan --- .../ai-ml/databases-and-ai.mdx | 7 - .../ai-ml/images/pgai-overview.png | 3 + advocacy_docs/edb-postgres-ai/ai-ml/index.mdx | 16 +- .../ai-ml/install-tech-preview.mdx | 105 ++++++++ .../edb-postgres-ai/ai-ml/overview.mdx | 27 +- .../edb-postgres-ai/ai-ml/postgres-and-ai.mdx | 7 - .../ai-ml/using-tech-preview/index.mdx | 10 + .../standalone-embedding-functions.mdx | 53 ++++ .../working-with-ai-data-in-S3.mdx | 89 +++++++ .../working-with-ai-data-in-postgres.mdx | 232 ++++++++++++++++++ advocacy_docs/edb-postgres-ai/index.mdx | 3 +- src/pages/index.js | 10 +- 12 files changed, 522 insertions(+), 40 deletions(-) delete mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/images/pgai-overview.png create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/install-tech-preview.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/standalone-embedding-functions.mdx create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-S3.mdx create mode 100644 advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-postgres.mdx diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx deleted file mode 100644 index 86cf7a76b59..00000000000 --- a/advocacy_docs/edb-postgres-ai/ai-ml/databases-and-ai.mdx +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: EDB Postgres AI AI/ML - Databases and AI -navTitle: Databases and AI -description: An introduction to the intersection of databases and AI/ML with EDB Postgres AI. ---- - -_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!_ diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/images/pgai-overview.png b/advocacy_docs/edb-postgres-ai/ai-ml/images/pgai-overview.png new file mode 100644 index 00000000000..1d82930782f --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/images/pgai-overview.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6202298c01b91caa7a8ea8b81fe3265cf4a9bc13bf6715ebc6008a92613211fd +size 67416 diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx index 3be6994c1a0..f056c6dff8f 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/index.mdx @@ -5,10 +5,18 @@ indexCards: simple iconName: BrainCircuit navigation: - overview -- databases-and-ai -- postgres-and-ai +- install-tech-preview +- using-tech-preview --- -EDB Postgres AI AI/ML. +Postgres AI Database is designed to solve all AI data management needs, including storing, searching, and retrieving of AI data. This uplevels Postgres to a database that manages and serves all types of data modalities directly and combines it with its battle-proof strengths as an established Enterprise system of record that manages high-value business data. + +In this tech preview, you will be able to use the pgai extension to build a simple retrieval augmented generation (RAG) application in Postgres. + +An [overview](overview) of the pgai extension gives you a high-level understanding of the major functionality available to date. + +To get started, you will need to [install the pgai tech preview](install-tech-preview) and then you can start [using the pgai tech preview](using-tech-preview) to build your RAG application. + + + -_EDB Postgres AI AI/ML is a set of tools, utilities and extensions that are designed to help you build and deploy machine learning and artificial intelligence models on your EDB Postgres AI databases. Building machine learning and artificial intelligence models on your EDB Postgres AI databases has never been easier._ \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/install-tech-preview.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/install-tech-preview.mdx new file mode 100644 index 00000000000..89535cad785 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/install-tech-preview.mdx @@ -0,0 +1,105 @@ +--- +title: EDB Postgres AI AI/ML - Installing the pgai tech preview +navTitle: Installing +description: How to install the EDB Postgres AI AI/ML pgai tech preview and run the container image. +prevNext: true +--- + +The preview release of pgai is distributed as a self-contained Docker container that runs PostgreSQL and includes all of the pgai dependencies. + +## Configuring and running the container image + +If you haven’t already, sign up for an EDB account and log in to the EDB container registry. +Login to docker with your the username tech-preview and your EDB Repo 2.0 Subscription Token as your password: + +``` +docker login docker.enterprisedb.com -u tech-preview -p +__OUTPUT__ +Login Succeeded +``` + +Download the pgai container image: + +``` +docker pull docker.enterprisedb.com/tech-preview/pgai +__OUTPUT__ +... +Status: Downloaded newer image for docker.enterprisedb.com/tech-preview/pgai:latest +docker.enterprisedb.com/tech-preview/pgai:latest +``` + +Specify a password to use for Postgres in the environment variable PGPASSWORD. The tech preview container will set up Postgres with this password and use it to connect to it. In bash or zsh set it as follows: + +``` +export PGPASSWORD= +``` + +You can use the pgai extension with encoder LLMs in Open AI or with open encoder LLMs from HuggingFace. If you want to use Open AI you also must provide your API key for that in the OPENAI_API_KEY environment variable: + +``` +export OPENAI_API_KEY= +``` + +You can use the pgai extension with AI data stored in Postgres tables or on S3 compatible object storage. To work with object storage you need to specify the ACCESS_KEY and SECRET_KEY environment variables:. + +``` +export ACCESS_KEY= +export SECRET_KEY= +``` + +Start the pgai tech preview container with the following command. It makes the tech preview PostgreSQL database available on local port 15432: + +``` +docker run -d --name pgai \ + -e ACCESS_KEY=$ACCESS_KEY \ + -e SECRET_KEY=$SECRET_KEY \ + -e OPENAI_API_KEY=$OPENAI_API_KEY \ + -e POSTGRES_PASSWORD=$PGPASSWORD \ + -e PGDATA=/var/lib/postgresql/data/pgdata \ + -p 15432:5432 \ + docker.enterprisedb.com/tech-preview/pgai:latest +``` + + +## Connect to Postgres + +If you haven’t yet, install the Postgres command-line tools. If you’re on a Mac, using Homebrew, you can install it as follows: + +``` +brew install libpq +``` + +Connect to the tech preview PostgreSQL running in the container. Note that this relies on $PGPASSWORD being set - if you’re using a different terminal for this part, make sure you re-export the password: + +``` +$ psql -h localhost -p 15432 -U postgres postgres +__OUTPUT__ +psql (16.1, server 16.3 (Debian 16.3-1.pgdg120+1)) +Type "help" for help. + +postgres=# +``` + + +Install the pgai extension: + +```sql +postgres=# create extension pgai cascade; +__OUTPUT__ +NOTICE: installing required extension "plpython3u" +NOTICE: installing required extension "vector" +CREATE EXTENSION +``` + +```sql +postgres=# \dx +__OUTPUT__ + List of installed extensions + Name | Version | Schema | Description +------------+---------+------------+------------------------------------------------------ + pgai | 0.0.1 | public | An extension to do the AIs + plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language + plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language + vector | 0.6.0 | public | vector data type and ivfflat and hnsw access methods +(4 rows) +``` diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx index 67660fa9e32..24c6765b802 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/overview.mdx @@ -1,32 +1,27 @@ --- title: EDB Postgres AI AI/ML - Overview navTitle: Overview -description: Where to start with EDB Postgres AI AI/ML, from creating innovative models to putting them into production. +description: Where to start with EDB Postgres AI AI/ML and the pgai tech preview. +prevNext: True --- -At the heart of EDB Postgres AI is the EDB Postgres AI database. This builds on Postgres's flexibility and extends its capability to include store the vector data of embeddings. +At the heart of EDB Postgres AI is the EDB Postgres AI database (pgai). This builds on Postgres's flexibility and extends its capability to include store the vector data of embeddings. -## What is vector data? +The pgai extension is currently available as a tech preview. It will be continuously extended with new functions. Here comes a description of the major functionality available to date. -To be able to deterimine similarity between things, we need to be able to represent them in a way that allows us to compare them. This is where vector data comes in, as it reduces the complexity of an thing to a set of numbers that can be compared. Those numbers could be derived from a set of images of the thing, a set of words about it or even a set of sounds. Each number exists on a dimension of the vector and vectors can have as many dimensions at they need to fully repreent a thing. +![PGAI Overview](images/pgai-overview.png) -## What are embeddings? +pgai introduced the concept of a “retriever” that you can create for a given type and location of AI data. Currently pgai supports unstructured plain text documents as well as a set of image formats. This data can either reside in regular columns of a Postgres table or it can reside in an S3 compatible object storage bucket. -These sets of values for the dimensions are called embeddings. Once we have embeddings for real world object, or images or text, we can compare them to see how similar they are across all the aspects represented by the embeddings dimensions. This comparison a core operation performed by machine learning and artificial intelligence models to work out connections between things. +A retriever encapsulates all processing that is needed to make the AI data in the provided source location searchable and retrievable via similarity. The application just needs to create a retriever via the pgai.create_retriever() function. When auto_embedding=TRUE is specified the pgai extension will automatically generate embeddings for all the data in the source location. Otherwise it will be up to the application to request a bulk generation of embeddings via pgai.refresh_retriever(). -## How are embeddings created? +Auto embedding is currently supported for AI data stored in Postgres tables and it automates the embedding updates using Postgres triggers. You can also combine the two options by using pgai.refresh_retriever() to embed all previously existing data and also setting `auto_embedding=TRUE` to generate embeddings for all new and changed data from now on. -Embeddings can be created manually, by experts who understand the thing to be represented. This is expensive and doesn't scale. The other option is for it to be created automatically by machine learning models, which can be initially trained by experts. These models are then trained on a set of data that represents the thing to be embedded, and then the model is used to generate the embeddings. The model is tuned by adjusting the values of the embeddings until the model can predict the thing accurately. - -## How are embeddings stored in a database? - -Embeddings, as sets of numbers, are not easily stored in a traditional relational database. They are best stored in a database that is designed to handle them. This is where EDB Postgres AI comes in. Building on the popular PG/Vector extension, EDB Postgres AI extends the capabilities of Postgres to handle vector data, and then provides the tools and utilties you need to build and deploy machine learning and artificial intelligence models. - -## What can I do with EDB Postgres AI? - -In the coming months, we'll be adding more content to this site to help you get started with EDB Postgres AI. We'll be covering how to create embeddings, how to train models and how to deploy them. We'll also be covering how to use the models in your applications, and how to monitor and manage them. We'll also be covering how to scale your models and how to use them in a production environment. +All embedding generation, storage, indexing and management is handled by the pgai extension internally. The application just has to specify the encoder LLM that the retriever should be using for this specific data and use case. +Once a retriever is created and all embeddings are up to date, the application can just use pgai.retrieve() to run a similarity search and retrieval by providing a query input. When the retriever is created for text data, the query input is also a text term. For image retrievers the query input is an image. The pgai retriever makes sure to use the same encoder LLM for the query input, conducts a similarity search and finally returns the ranked list of similar data from the source location. +pgai currently supports a broad list of open encoder LLMs from HuggingFace as well as a set of OpenAI encoders. Just consult the list of supported encoder LLMs in the pgai.encoders meta table. HuggingFace LLMs are running locally on the Postgres node, while OpenAI encoders involve a call out to the OpenAI cloud service. diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx deleted file mode 100644 index 1ca96dc4512..00000000000 --- a/advocacy_docs/edb-postgres-ai/ai-ml/postgres-and-ai.mdx +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: EDB Postgres AI AI/ML - Postgres and AI -navTitle: Postgres and AI -description: From the general to the specific, this section provides an overview of the intersection of Postgres and AI/ML with EDB Postgres AI. ---- - -_The EDB Postgres AI AI/ML official documentation is currently under development. Stay tuned for updates!_ diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx new file mode 100644 index 00000000000..7dec7824cca --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx @@ -0,0 +1,10 @@ +--- +title: EDB Postgres AI AI/ML - Using the pgai tech preview +navTitle: Using +description: Using the EDB Postgres AI AI/ML tech preview to build a simple retrieval augmented generation (RAG) application in Postgres. +navigation: +- working-with-ai-data-in-postgres +- working-with-ai-data-in-s3 +- standard-encoders +--- + diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/standalone-embedding-functions.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/standalone-embedding-functions.mdx new file mode 100644 index 00000000000..809273d4102 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/standalone-embedding-functions.mdx @@ -0,0 +1,53 @@ +--- +title: Stand-alone Embedding Functions in pgai +navTitle: Stand-alone Embedding Functions +description: Use the pgai extension to generate embeddings for images and text. +--- + +Use generate_single_image_embedding function to get embeddings for the given image. Currently, model_provider can only be openai or huggingface. You can check the list of valid embedding models and model providers from the Encoders Supported PGAI section. + +```sql +SELECT pgai.generate_single_image_embedding( + 'clip-vit-base-patch32', -- embedding model name + 'openai', -- model provider + 'https://s3.us-south.cloud-object-storage.appdomain.cloud', -- S3 endpoint + 'torsten', -- S3 bucket name + 'foto.jpg' -- img object to generate embeddings +); +__OUTPUT__ + generate_single_image_embedding +-------------------------------- + +(1 row) +``` + +Use the `generate_text_embedding` function to get embeddings for the given image. Currently, the `model_provider` can only be `openai` or `huggingface`. + +```sql +SELECT pgai.generate_text_embedding( +'text-embedding-3-small', -- embedding model name +'openai', -- model provider + 0, -- dimensions, setting 0 will replace with the default value in encoder's table +'Veggie Burger' -- text to generate embeddings +); +__OUTPUT__ + generate_text_embedding +------------------------ + +(1 row) +``` + + +You can check the list of valid embedding models and model providers from pgai.encoders table + +```sql +SELECT provider, count(*) encoder_model_count FROM pgai.encoders group by (provider); +__OUTPUT__ + provider | encoder_model_count +-------------+--------------------- + huggingface | 36 + openai | 4 +(2 rows) +``` + + diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-S3.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-S3.mdx new file mode 100644 index 00000000000..905359f81c8 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-S3.mdx @@ -0,0 +1,89 @@ +--- +title: Working with AI data stored in S3-compatible object storage +navTitle: Working with AI data in S3 +description: How to work with AI data stored in S3-compatible object storage using the pgai extension. +--- + +We recommend you to prepare your own S3 compatible object storage bucket with some test data and try the steps in this section with that. But it is possible to simply use the example S3 bucket data as is in the examples here even with your custom access key and secret key credentials because these have been configured for public access. + +In addition we use image data and an according image encoder LLM in this example instead of text data. But you could also use plain text data on object storage similar to the examples in the previous section. + +First let’s create a retriever for images stored on s3-compatible object storage as the source. We specify torsten as the bucket name and an endpoint URL where the bucket is created. We specify an empty string as prefix because we want all the objects in that bucket. We use the [`clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32) open encoder model for image data from HuggingFace. We provide a name for the retriever so that we can identify and reference it subsequent operations: + +```sql +SELECT pgai.create_s3_retriever( + 'image_embeddings', -- Name of the similarity retrieval setup + 'public', -- Schema of the source table + 'img_id', -- Primary key + 'clip-vit-base-patch32', -- Embeddings encoder model for similarity data + 'img', -- data type, could be either img or text + 'torsten', -- S3 bucket name + 'https://s3.us-south.cloud-object-storage.appdomain.cloud', -- s3 endpoint address + '' -- prefix +); +__OUTPUT__ + create_s3_retriever +--------------------- + +(1 row) +``` + + +Next, run the refresh_retriever function. + +```sql +SELECT pgai.refresh_retriever('image_embeddings'); +__OUTPUT__ + refresh_retriever +------------------- + +(1 row) +``` + +Finally, run the retrieve_via_s3 function with the required parameters to retrieve the top K most relevant (most similar) AI data items. Please be aware that the object type is currently limited to image and text files. + +```sql +SELECT data from pgai.retrieve_via_s3( + 'image_embeddings', -- retriever's name + 1, -- top K + 'torsten', -- S3 bucket name + 'foto.jpg', -- object name + 'https://s3.us-south.cloud-object-storage.appdomain.cloud' +); +__OUTPUT__ + data +-------------------- + {'img_id': 'foto'} + (1 row) +``` + +If you set the `ACCESS_KEY` and `SECRET_KEY` you can use the following queries to run without an `s3_endpoint` setting using a command like this: + +```sql +SELECT pgai.create_s3_retriever( + 'img_file_embeddings', -- Name of the similarity retrieval setup + 'demo', -- Schema of the source table + 'img_id', -- Primary key + 'clip-vit-base-patch32', -- Embeddings encoder model for similarity data + 'img', -- data type + 'bilge-ince-test' -- S3 bucket name +); +__OUTPUT__ + create_s3_retriever +--------------------- + +(1 row) +``` + +```sql +SELECT pgai.refresh_retriever('img_file_embeddings'); +``` + +```sql +SELECT data from pgai.retrieve_via_s3('img_file_embeddings', + 1, + 'bilge-ince-test', + 'kirpis_small.jpg', + 'http://s3.eu-central-1.amazonaws.com' + ); +``` diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-postgres.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-postgres.mdx new file mode 100644 index 00000000000..f0af183386a --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/working-with-ai-data-in-postgres.mdx @@ -0,0 +1,232 @@ +--- +title: Working with AI data stored in Postgres tables +navTitle: Working with AI data in Postgres +description: How to work with AI data stored in Postgres tables using the pgai extension. +--- + +We will first look at working with AI data stored in columns in the Postgres table. + +To see how to use AI data stored in S3-compatible object storage, please skip to the next section. + +First let’s create a Postgres table for some test AI data: + +```sql +CREATE TABLE products ( + product_id SERIAL PRIMARY KEY, + product_name TEXT NOT NULL, + description TEXT, + last_updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP +); +__OUTPUT__ +CREATE TABLE +``` + + +Now let’s create a retriever with the just created products table as the source. We specify product_id as the unique key column to and we define the product_name and description columns to use for the similarity search by the retriever. We use the text-embeddings-3-small open encoder model from HuggingFace. We set auto_embedding to True so that any future insert, update or delete to the source table will automatically generate, update or delete also the corresponding embedding. We provide a name for the retriever so that we can identify and reference it subsequent operations: + +```sql +SELECT pgai.create_pg_retriever( + 'product_embeddings_auto', -- Retriever name + 'public', -- Schema + 'product_id', -- Primary key + 'text-embedding-3-small', -- embedding model + 'text', -- data type + 'products', -- Source table + ARRAY['product_name', 'description'], -- Columns to vectorize + TRUE -- auto embeddings TRUE to set trigger +); +__OUTPUT__ + create_pg_retriever +--------------------- + +(1 row) +``` + + + +Now let’s insert some AI data records into the products table. Since we have set auto_embedding to True, the retriever will automatically generate all embeddings in real-time for each inserted record: + +```sql +INSERT INTO products (product_name, description) VALUES + ('Hamburger', 'Tasty'), + ('Cheesburger', 'Very tasty'), + ('Fish n Chips', 'Naa'), + ('Fries', 'Dunno'), + ('Burrito', 'Always'), + ('Pizza', 'Mkay'), + ('Sandwich', 'So what'), + ('Veggie Burger', 'Go away'), + ('Kebab', 'Maybe'); +__OUTPUT__ +INSERT 0 9 +``` + +Now we can directly use the retriever (specifying the retriever name) for a similarity retrieval of the top K most relevant (most similar) AI data items: + +```sql +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_auto' -- retriever's name +); +__OUTPUT__ + data +------------------------------------- + {'data': 'Hamburger - Tasty'} + {'data': 'Cheesburger - Very tasty'} + {'data': 'Fries - Dunno'} + {'data': 'Sandwich - So what'} + {'data': 'Kebab - Maybe'} +(5 rows) +``` + +Now let’s try a retriever without auto embedding. This means that the application has control over when the embeddings are computed in a bulk fashion. For demonstration we can simply create a second retriever for the same products table that we just created above: + +```sql +SELECT pgai.create_pg_retriever( + 'product_embeddings_bulk', -- Retriever name + 'public', -- Schema + 'product_id', -- Primary key + 'text-embedding-3-small', -- embedding model + 'text', -- data type + 'products', -- Source table + ARRAY['product_name', 'description'], -- Columns to vectorize + FALSE -- auto embeddings FALSE +); +__OUTPUT__ + create_pg_retriever +--------------------- + +(1 row) +``` + + +We created this second retriever on the products table after we have inserted the AI records there. If we run a retrieve operation now we would not get back any results: + +``` +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_bulk' -- retriever's name +); +__OUTPUT__ + data +------ +(0 rows) +``` + +That’s why we first need to run a bulk generation of embeddings. This is achieved via the `refresh_retriever()` function: + +``` +SELECT pgai.refresh_retriever( +'product_embeddings_bulk' -- name of the retriever +); +__OUTPUT__ +INFO: inserted table name public._pgai_embeddings_product_embeddings_bulk + refresh_retriever +------------------- + +(1 row) +``` + +Now we can run the same retrieve operation with the second retriever as above: + +```sql +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_bulk' -- retriever's name +); +__OUTPUT__ + data +------------------------------------- + {'data': 'Hamburger - Tasty'} + {'data': 'Cheesburger - Very tasty'} + {'data': 'Fries - Dunno'} + {'data': 'Sandwich - So what'} + {'data': 'Kebab - Maybe'} +(5 rows) +``` + +Now let’s see what happens if we add additional AI data records: + +```sql +INSERT INTO products (product_name, description) VALUES + ('Chicken Nuggets', 'Never'), + ('Ramen', 'Delicious'); +__OUTPUT__ +INSERT 0 2 +``` + +The new data is automatically picked up in the retrieval from the first retriever with auto embeddings: + +```sql +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_auto' -- retriever's name +); +__OUTPUT__ + data +-------------------------------------- + {'data': 'Hamburger - Tasty'} + {'data': 'Cheesburger - Very tasty'} + {'data': 'Sandwich - So what'} + {'data': 'Kebab - Maybe'} + {'data': 'Ramen - Delicious'} +(5 rows) +``` + +At the same time the second retriever without auto embedding does not reflect the new data until there is another explicit refresh_retriever() run: + +```sql +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_bulk' -- retriever's name +); +__OUTPUT__ + data +------------------------------------- + {'data': 'Hamburger - Tasty'} + {'data': 'Cheesburger - Very tasty'} + {'data': 'Fries - Dunno'} + {'data': 'Sandwich - So what'} + {'data': 'Kebab - Maybe'} +(5 rows) +``` + +If we now call `refresh_retriever()` again, the new data is picked up: + +```sql +SELECT pgai.refresh_retriever( +'product_embeddings_bulk' -- name of the retriever +); +__OUTPUT__ +INFO: inserted table name public._pgai_embeddings_product_embeddings_bulk + refresh_retriever +------------------- +``` + +And will be returned when we run the retrieve operation again: + +```sql +SELECT data FROM pgai.retrieve( +'I like it', -- The query text to retrieve the top similar data + 5, -- top K +'product_embeddings_bulk' -- retriever's name +); +__OUTPUT__ + data +-------------------------------------- + {'data': 'Hamburger - Tasty'} + {'data': 'Cheesburger - Very tasty'} + {'data': 'Sandwich - So what'} + {'data': 'Kebab - Maybe'} + {'data': 'Ramen - Delicious'} +(5 rows) +``` + +We used the two different retrievers for the same source data just to demonstrate the workings of auto embedding compared to explicit `refresh_retriever()`. In practice you may want to combine auto embedding and refresh_retriever() in a single retriever to conduct an initial embedding of data that existed before you created the retriever and then rely on auto embedding for any future data that is ingested, updated or deleted. + +You should consider relying on refresh_retriever() only, without auto embedding, if you typically ingest a lot of AI data at once in batch manner. diff --git a/advocacy_docs/edb-postgres-ai/index.mdx b/advocacy_docs/edb-postgres-ai/index.mdx index 708c7faffbd..c86c0cd9a24 100644 --- a/advocacy_docs/edb-postgres-ai/index.mdx +++ b/advocacy_docs/edb-postgres-ai/index.mdx @@ -4,7 +4,8 @@ navTitle: Home directoryDefaults: product: "EDB Postgres AI" iconName: edb_postgres_ai/LoopBlack - indexCards: full + indexCards: simple + prevNext: true navigation: - overview - console diff --git a/src/pages/index.js b/src/pages/index.js index 0a1fcca3d98..a721f386df8 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -215,13 +215,13 @@ const Page = () => { to="/edb-postgres-ai/ai-ml" > - Overview + Overview of pgai - - Databases and AI + + Install the Tech Preview - - Postgres and AI + + Use the Tech Preview Date: Wed, 22 May 2024 14:05:44 -0700 Subject: [PATCH 47/67] Fix some links --- src/pages/index.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/index.js b/src/pages/index.js index a721f386df8..385551f1a2a 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -217,10 +217,10 @@ const Page = () => { Overview of pgai - + Install the Tech Preview - + Use the Tech Preview From 4ed229d2c37e01c1df10fc8c3ec296eb41551774 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 22 May 2024 22:14:10 +0100 Subject: [PATCH 48/67] Added using intro Signed-off-by: Dj Walker-Morgan --- .../edb-postgres-ai/ai-ml/using-tech-preview/index.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx index 7dec7824cca..981d119b782 100644 --- a/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx +++ b/advocacy_docs/edb-postgres-ai/ai-ml/using-tech-preview/index.mdx @@ -8,3 +8,9 @@ navigation: - standard-encoders --- +This section shows how you can use your [newly installed pgai tech preview](install-tech-preview) to retrieve and generate AI data in Postgres. + +* [Working with AI data in Postgres](working-with-ai-data-in-postgres) details how to use the pgai extension to work with AI data stored in Postgres tables. +* [Working with AI data in S3](working-with-ai-data-in-s3) covers how to use the pgai extension to work with AI data stored in S3 compatible object storage. +* [Standard encoders](standard-encoders) goes through the standard encoder LLMs that are supported by the pgai extension. + From 41c74c73306a87a910bea80fb257872e009d34e8 Mon Sep 17 00:00:00 2001 From: gvasquezvargas Date: Wed, 22 May 2024 18:55:55 +0200 Subject: [PATCH 49/67] Added cross-linking and AI Copilot info --- advocacy_docs/edb-postgres-ai/tools/backup.mdx | 2 +- advocacy_docs/edb-postgres-ai/tools/index.mdx | 2 +- advocacy_docs/edb-postgres-ai/tools/management.mdx | 4 ++-- .../edb-postgres-ai/tools/migration-and-ai.mdx | 14 +++++++++----- 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/tools/backup.mdx b/advocacy_docs/edb-postgres-ai/tools/backup.mdx index 023a7f66927..bbe47b016ee 100644 --- a/advocacy_docs/edb-postgres-ai/tools/backup.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/backup.mdx @@ -4,7 +4,7 @@ navTitle: Backup and Recovery description: The backup and recovery tools available in EDB Postgres AI Tools --- -Barman is a tool for managing backup and recovery of PostgreSQL databases. It is designed for business critical databases and provides features such as backup catalogues, incremental backup, retention policies, and remote recovery. +[Barman](/supported-open-source/barman/) is a tool for managing backup and recovery of PostgreSQL databases. It is designed for business critical databases and provides features such as backup catalogues, incremental backup, retention policies, and remote recovery. Barman is integrated with EDB Postgres AI and can be used to manage backups of your EDB Postgres AI databases. diff --git a/advocacy_docs/edb-postgres-ai/tools/index.mdx b/advocacy_docs/edb-postgres-ai/tools/index.mdx index 3c4b4c8779c..4ea2b294839 100644 --- a/advocacy_docs/edb-postgres-ai/tools/index.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/index.mdx @@ -13,5 +13,5 @@ EDB Postgres AI Tools Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. -_EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered._ +EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered. diff --git a/advocacy_docs/edb-postgres-ai/tools/management.mdx b/advocacy_docs/edb-postgres-ai/tools/management.mdx index 683640daeab..ff5a1924152 100644 --- a/advocacy_docs/edb-postgres-ai/tools/management.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/management.mdx @@ -6,6 +6,6 @@ description: An introduction to the management tools of EDB Postgres AI such as An introduction to the management tools of EDB Postgres AI such as Postgres Enterprise Manager. -PEM is a comprehensive management tool for EDB Postgres Advanced Server and PostgreSQL databases. PEM provides database administrators with a graphical view of the server, allowing them to easily monitor and manage their databases. PEM also provides tools for database design, monitoring, and tuning, as well as tools for managing database objects, users, and roles. +[PEM](/pem/latest/) is a comprehensive management tool for EDB Postgres Advanced Server and PostgreSQL databases. PEM provides database administrators with a graphical view of the server, allowing them to easily monitor and manage their databases. PEM also provides tools for database design, monitoring, and tuning, as well as tools for managing database objects, users, and roles. -PEM is a web-based application that can be accessed from any web browser. PEM provides a single pane of glass for managing multiple database servers, allowing administrators to easily monitor and manage their databases from a centralized location. \ No newline at end of file +PEM is a web-based application that can be [accessed from any web browser](/pem/latest/pem_web_interface/). PEM provides a single pane of glass for managing multiple database servers, allowing administrators to easily monitor and manage their databases from a centralized location. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx b/advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx index 662be27c24d..92d0141a3ec 100644 --- a/advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/migration-and-ai.mdx @@ -4,10 +4,14 @@ navTitle: Migration and AI description: The Migration offering of EDB Postgres AI Tools includes an innovative migration copilot. --- -EDB Postgres AI Tools Migration Portal already offeres an AI copilot to assist users who are migration their databases to EDB Postgres. The AI copilot is a chatbot that helps users with the migration process. The AI copilot is designed to help users with the following tasks: +EDB Postgres AI Tools Migration Portal offers an [AI copilot](/migration_portal/latest/03_mp_using_portal/mp_ai_copilot/) to assist users who are migrating their databases to EDB Postgres. +The AI copilot is an AI-driven chatbot tool that helps users with the migration process. +The AI copilot is designed to help users with the following tasks: -- **Migration Assessment**: The AI copilot can help users assess their migration readiness. -- **Migration Planning**: The AI copilot can help users plan their migration. -- **General Migration Assistance**: The AI copilot can help users with general migration questions. +- **General Migration Assistance**: The AI copilot can help users with general migration questions. + For example, users can request information about available tools, and source and target database compatibility. +- **Migration Planning**: The AI copilot can help users plan their migration, and obtain an overview of the end-to-end migration paths. +- **Migration Assessment**: The AI copilot can help users assess their migration readiness. + For example, if there are compatibility issues between source and target databases, the AI Copilot can suggest compatible query alternatives. -The AI copilot is designed to be user-friendly and easy to use. Users can interact with the AI copilot using natural language. The AI copilot is also designed to be context-aware, so it can provide users with relevant information based on the context of the conversation. \ No newline at end of file +The AI copilot is designed to be user-friendly and easy to use. Users can interact with the AI copilot using natural language and improve the quality of answers with [good prompting](/migration_portal/latest/03_mp_using_portal/mp_ai_copilot/ai_good_prompts/). The AI copilot is also designed to be context-aware, so it can provide users with relevant information based on the context of the conversation. \ No newline at end of file From 20be7a55773b22b87dca038173eadd1a5a8cbdfe Mon Sep 17 00:00:00 2001 From: Josh Heyer Date: Wed, 22 May 2024 13:52:56 -0700 Subject: [PATCH 50/67] Update link to reflect actual path --- src/pages/index.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/index.js b/src/pages/index.js index 385551f1a2a..bd1bcdb01cd 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -229,7 +229,7 @@ const Page = () => { headingText="Tools" to="/edb-postgres-ai/tools" > - + Migration and AI From 66237ecf98b93791bc11df6880cd81cbf09ee5ec Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 09:08:33 +0100 Subject: [PATCH 51/67] Remove spare title Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/tools/index.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/tools/index.mdx b/advocacy_docs/edb-postgres-ai/tools/index.mdx index 4ea2b294839..f93637e4d74 100644 --- a/advocacy_docs/edb-postgres-ai/tools/index.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/index.mdx @@ -9,8 +9,6 @@ navigation: - backup --- -EDB Postgres AI Tools - Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered. From 236fa281377b813a9b93b0d8360bd63f77fda191 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 00:22:58 +0100 Subject: [PATCH 52/67] overview and database fixes Signed-off-by: Dj Walker-Morgan --- .../databases/cloudservice.mdx | 7 +- .../edb-postgres-ai/databases/databases.mdx | 20 ++++ .../edb-postgres-ai/databases/index.mdx | 6 +- .../edb-postgres-ai/databases/onprem.mdx | 10 -- .../edb-postgres-ai/databases/options.mdx | 25 ++++- .../edb-postgres-ai/overview/features.mdx | 11 --- .../edb-postgres-ai/overview/guide.mdx | 24 +++++ .../edb-postgres-ai/overview/index.mdx | 8 +- .../edb-postgres-ai/overview/releasenotes.mdx | 98 +++++++++++++++++++ src/pages/index.js | 12 +-- 10 files changed, 185 insertions(+), 36 deletions(-) create mode 100644 advocacy_docs/edb-postgres-ai/databases/databases.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/databases/onprem.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/overview/features.mdx create mode 100644 advocacy_docs/edb-postgres-ai/overview/releasenotes.mdx diff --git a/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx b/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx index 3936387d141..fd190cc3da8 100644 --- a/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx @@ -4,6 +4,9 @@ navTitle: Cloud Service description: An introduction to the EDB Postgres AI Cloud Service and its features. --- -Introduction to the EDB Postgres AI Cloud Service +The EDB Postgres AI Cloud Service, formerly known as BigAnimal is an evolution of the service to offer a holistic platform which offers hybrid data estate management, observability, and AI capabilities. + +The EDB Postgres AI Cloud Service itself is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for AI and machine learning workloads. It also allows provides the platform for EDB Postgres AI Analytics and EDB Postgres AI Machine Learning services. + +The service is built on the EDB Postgres Advanced Server and EDB Postgres Extended databases and is designed to help organizations accelerate the development and deployment of AI and machine learning applications. -_The EDB Postgres AI Cloud Service is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for AI and machine learning workloads. The service is built on the EDB Postgres Advanced Server database and is designed to help organizations accelerate the development and deployment of AI and machine learning applications._ diff --git a/advocacy_docs/edb-postgres-ai/databases/databases.mdx b/advocacy_docs/edb-postgres-ai/databases/databases.mdx new file mode 100644 index 00000000000..1d47ce64416 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/databases/databases.mdx @@ -0,0 +1,20 @@ +--- +title: EDB Postgres AI Databases +navTitle: Databases +description: Deploy EDB Postgres AI Databases on-premises with the EDB Postgres AI Estate and Agent components. +--- + +EDB Postgres databases are the core of the EDB Postgres AI platform. EDB Postgres Databases are available for self-managed deployment and on the EDB Postgres AI Cloud Service. Self-managed EDB Postgres databases can be integrated with the EDB Postgres AI Cloud Service estate and managed through a single pane of glass by installing the [EDB Postgres AI Agent](/edb-postgres-ai/console/agent). + +## EDB Postgres Advanced Server (EPAS) + +EDB Postgres Advanced Server (EPAS) is an enhanced version of PostgreSQL that is designed to meet the needs of large-scale, mission-critical enterprise workloads. EPAS is built on the open source PostgreSQL database, and includes additional enterprise-class features and capabilities that are critical for enterprise database deployments. These include Oracle compatibility and transparent data encryption. EPAS is available for self-managed deployment and on the EDB Postgres AI Cloud Service. + +## EDB Postgres Extended Server (PGE) + +EDB Postgres Extended Server (PGE) is an enhanced version of PostgreSQL that is designed to meet the needs of large-scale, mission-critical enterprise workloads. PGE is built on the open source PostgreSQL database, and includes additional enterprise-class features and capabilities that are critical for enterprise database deployments. This includes transparent data encryption. PGE is available for self-managed deployment and on the EDB Postgres AI Cloud Service. + +## EDB Postgres Distributed + +EDB Postgres Distributed (PGD) is a high availability solution for EDB Postgres databases. PGD provides a distributed database environment that is designed to ensure high availability and fault tolerance for mission-critical workloads. PGD can be used with EPAS, PGE or PostgreSQL databases. PGD is available for self-managed deployment and on the EDB Postgres AI Cloud Service (as the Distributed High Availability option). + diff --git a/advocacy_docs/edb-postgres-ai/databases/index.mdx b/advocacy_docs/edb-postgres-ai/databases/index.mdx index aa56a6733d1..01e75499804 100644 --- a/advocacy_docs/edb-postgres-ai/databases/index.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/index.mdx @@ -5,9 +5,11 @@ indexCards: simple iconName: Database --- -Introduction to the EDB Postgres AI Databases and the EDB Postgres AI Cloud Service +Building on decades of Postgres expertise, the EDB Postgres databases are the core of the EDB Postgres AI platform. EDB Postgres Advanced Server can take on Oracle workloads, while EDB Postgres Extended Server is designed for large-scale, mission-critical enterprise workloads. EDB Postgres Distributed provides high availability and fault tolerance for mission-critical workloads. -_The EDB Postgres AI Databases are a set of databases that are optimized for machine learning and artificial intelligence workloads. The databases are designed to provide high performance and scalability for machine learning and artificial intelligence workloads. The EDB Postgres AI Databases are available as part of the EDB Postgres AI Cloud Service._ +For here you can read more about the [databases](databases) that power EDB Postgres AI, and how they can be deployed on-premises with the EDB Postgres AI Estate and Agent components. +You can also learn about the [EDB Postgres AI Cloud Service](cloudservice) and how it can be used to manage your database estate. +Finally, there's an outline of some of the options available when deploying EDB Postgres databases. diff --git a/advocacy_docs/edb-postgres-ai/databases/onprem.mdx b/advocacy_docs/edb-postgres-ai/databases/onprem.mdx deleted file mode 100644 index 187c4977150..00000000000 --- a/advocacy_docs/edb-postgres-ai/databases/onprem.mdx +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: EDB Postgres AI Databases- On Premises -navTitle: On Premises -description: Deploy EDB Postgres AI Databases on-premises with the EDB Postgres AI Estate and Agent components. ---- - -EDB Postgres AI Databases are available for on-premises deployment. - -_The on-premises deployment of EDB Postgres AI Database are a self-managed deployment, which can use the EDB Postgres AI Estate and Agent components to manage and integrate the on-premises deployment alongside the EDB Postgres AI Database Cloud Service through the EDB Postgres AI Console's single pane of glass._ - diff --git a/advocacy_docs/edb-postgres-ai/databases/options.mdx b/advocacy_docs/edb-postgres-ai/databases/options.mdx index e51101919a7..cd28fbf38f9 100644 --- a/advocacy_docs/edb-postgres-ai/databases/options.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/options.mdx @@ -1,10 +1,27 @@ --- -title: EDB Postgres AI Databases - Optional Features -navTitle: Optional Features +title: EDB Postgres AI Databases - Deployment Options +navTitle: Deployment Options description: High Availability and other available for EDB Postgres AI Databases and on EDB Postgres AI Cloud Service. +deepToC: true --- -Introducing EDB Postgres Distributed and other optional features available for EDB Postgres AI Databases. +## Availablitity Options -_EDB Postgres Distributed is a separate product that is available for EDB Postgres AI Databases. It provides a distributed SQL database that is designed to support high availability. EDB Postgres Distributed powers the EDB Postgres AI Cloud Service's distributed high availability option._ +### Single Instance + +Single instance databases are great for development and testing, but for production workloads, you need to consider high availability and fault tolerance. + +### Primary/Secondary Replication + +Primary/Secondary replication is a common high availability solution for databases. In this configuration, a primary database server is responsible for processing read and write requests. A secondary database server is configured to replicate the primary database server. If the primary database server fails, the secondary database server can take over and become the primary database server. + +This configuration provides fault tolerance and high availability in a particular location. This can be used with EDB Postgres Advanced Server (EPAS) and EDB Postgres Extended Server (PGE). + +This is a standard configuration option on EDB Postgres AI Cloud Service. + +## Distributed High Availability + +High availability is a critical requirement for mission-critical workloads. EDB Postgres Distributed (PGD) provides a distributed database environment that is designed to ensure high availability and fault tolerance for mission-critical workloads. PGD can be used with EPAS, PGE or PostgreSQL databases. PGD is available for self-managed deployment and on the EDB Postgres AI Cloud Service (as the Distributed High Availability option). + +This is also a standard configuration option on EDB Postgres AI Cloud Service. diff --git a/advocacy_docs/edb-postgres-ai/overview/features.mdx b/advocacy_docs/edb-postgres-ai/overview/features.mdx deleted file mode 100644 index c6817877d57..00000000000 --- a/advocacy_docs/edb-postgres-ai/overview/features.mdx +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: EDB Postgres AI Overview - Features -navTitle: Features -description: The current features of EDB Postgres AI, element by element. ---- - -What are the features of EDB Postgres AI? This section provides a detailed look at the features of EDB Postgres AI, element by element. - -* _ -* _ -* _ diff --git a/advocacy_docs/edb-postgres-ai/overview/guide.mdx b/advocacy_docs/edb-postgres-ai/overview/guide.mdx index d2d08853c95..b4ba0f3f3bc 100644 --- a/advocacy_docs/edb-postgres-ai/overview/guide.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/guide.mdx @@ -4,3 +4,27 @@ navTitle: Guide description: What do you want to use EDB Postgres AI for? Start navigating the documentation here. --- +## Do you want to run analytical queries on your Postgres data? + +You'll want to look at the [EDB Postgres AI Analytics](/edb-postgres-ai/analytics) documentation, which covers the Postgres Lakehouse. + +## Are you looking at running machine learning models on your Postgres data? + +You'll want to look at the [EDB Postgres AI Machine Learning](/edb-postgres-ai/ai-ml) documentation, which covers the technical preview of the pgai extension. + +## Do you need to migrate your data to Postgres? + +You'll want to look at the [EDB Postgres AI Migration](/edb-postgres-ai/migration) documentation, which covers the Migration Toolkit and Migration Portal. + +## Do you want to manage your EDB Postgres AI Cloud databases? + +You'll want to look at the [EDB Postgres AI Console](/edb-postgres-ai/console) documentation, which covers the Console and Cloud databases. + +## Do you want to manage your self-managed Postgres databases as part of a hybrid data estate? + +You'll want to look at the [EDB Postgres AI Platform Agent](/ebd-postgres-ai/console/agent) documentation, which covers the Estate management and on premises agent installation. + +## Do you want to know more about the EDB Postgres AI Cloud Service? + +You'll want to look at the [EDB Postgres AI Cloud Service](/edb-postgres-ai/cloud-service) documentation, which covers the Cloud Service and its databases. + diff --git a/advocacy_docs/edb-postgres-ai/overview/index.mdx b/advocacy_docs/edb-postgres-ai/overview/index.mdx index c425177a694..8e649a63560 100644 --- a/advocacy_docs/edb-postgres-ai/overview/index.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/index.mdx @@ -3,9 +3,15 @@ title: EDB Postgres AI Overview navTitle: Overview indexCards: simple iconName: Earth +deepToC: true --- -The overview of EDB Postgres AI provides a wide-angle view of the product and its capabilities. This section includes the following: +EDB Postgres AI is a new era for EDB. With EDB Postgres AI, customers can now leverage EDB’s enterprise-grade Postgres offerings to support not just their mission critical transactional workloads, but also their analytical and AI applications. This also means that, in addition to the core transactional database releases you have come to expect from EDB, we will now be delivering regular updates to our analytics, AI, and platform capabilities. +In this overview section we will: + +* [Introduce the concepts that underpin EDB Postgres AI](concepts) +* [Provide a guide to help you navigate the documentation](guide) +* [Share the latest features released and updated in EDB Postgres AI](releasenotes) diff --git a/advocacy_docs/edb-postgres-ai/overview/releasenotes.mdx b/advocacy_docs/edb-postgres-ai/overview/releasenotes.mdx new file mode 100644 index 00000000000..f4a02416537 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/overview/releasenotes.mdx @@ -0,0 +1,98 @@ +--- +title: EDB Postgres AI Overview - Release Notes +navTitle: Release Notes +description: The current features released and updated in EDB Postgres AI. +--- + +EDB Postgres AI is a a new era for EDB. With EDB Postgres AI, customers can now +leverage EDB’s enterprise-grade Postgres offerings to support not just their +mission critical transactional workloads, but also their analytical and AI +applications. This also means that, in addition to the core transactional +database releases you have come to expect from EDB, we will now be delivering +regular updates to our analytics, AI, and platform capabilities. + +This is the first quarterly release of EDB Postgres AI, delivering key +functionality to support the platform's vision. This release includes analytical +and vector database capabilities, single pane of glass management and +observability for hybrid data estates, and an AI migration copilot. + +## EDB Analytics and AI Updates + +Customers can now launch Postgres Lakehouse nodes using the EDB Postgres AI +Cloud Service (formerly EDB BigAnimal) to get results of analytical (OLAP) +queries much faster than from typical transactional Postgres. Postgres Lakehouse +nodes are available now for customers using EDB Postgres AI - Hosted +environments on AWS, and will be rolling out to additional cloud environments +soon. + +Postgres Lakehouse uses Apache DataFusion’s vectorized SQL query engine to +execute analytical queries 5-100x faster (30x on average) compared to native +Postgres, while still falling back to native execution when necessary. Postgres +Lakehouse nodes run either EPAS or PGE as the Postgres engine, with data for +analytics stored as columnar tables in object storage using the open source +Delta Lake protocol. Customers can sync tables from transactional sources +(initially, EDB Postgres AI Cloud Service databases) into Lakehouse Tables in +Managed Storage Locations (initially, S3 object storage buckets). + +### Technical Preview of [EDB pgai extension](/edb-postgres-ai/ai-ml) + +Customers can now access a technical preview of the new EDB pgai extension, +which seamlessly integrates and manages AI data for enterprise workloads with +EDB Postgres AI, to help understand your AI data directly out of the box. Built +on top of Postgres vector data support, this tech preview enables Postgres to +run LLMs and directly manage, process, search and retrieve AI data such as text +documents or images to accelerate AI application development and +operationalization across your company. + +In this technical preview, you'll have the opportunity to explore the pgai extension +and build AI-infused similarity search applications — for instance, a +Retrieval-Augmented Generation (RAG) application using Postgres. RAG +applications utilize a powerful combination of retrieval systems and language +models to provide accurate and context-aware responses to user queries. Learn +more and enroll in the tech preview [here](https://info.enterprisedb.com/pgai-preview). + +## EDB Platform updates + +### [EDB Postgres AI Platform Agent](/edb-postgres-ai/console) release and platform support + +As part of its initial release, the EDB Postgres AI agent enables users to +connect self-managed Postgres deployments to the platform, to enable unified +observability and management over hybrid data estates. Additionally, users will +be provided with the Postgres database version and size (in MB) in the EDB +Postgres AI Platform interface, with data collected from each database at a +configurable level. Additionally, EDB Postgres All Platform is available on +EDB-supported x86 Linux distros. + +## [EDB Postgres AI Database](/edb-postgres-ai/databases) updates + +### EDB Database Server updates + +As part of EDB’s support for the open source community’s quarterly release +schedule, we completed PGE and EPAS merge updates from the latest upstream +PostgreSQL, including the following: + +| Database Distributions | Versions Supported | +|------------------------------|------------------------------------------------| +| PostgreSQL | 16.3, 15.7, 14.12, 13.15 and 12.19 | +| EDB Postgres Extended Server | 16.3.0, 15.7.0, 14.12, 13.15 and 12.19 | +| EDB Postgres Advanced Server | 16.3.0, 15.7.0, 14.12.0, 13.15.21 and 12.19.24 | + +### EDB Postgres® Distributed 5.5 Release Enhancements + +#### Read scalability enhancements +EDB Postgres Distributed users can now increase client application performance +by spreading their read load across multiple nodes within a region. As a result, +enterprises realize better support of read-heavy workloads by routing their read +queries to a separate endpoint. The feature improves the former EDB Postgres +Distributed process, where client applications could only use the lead node to +route their application traffic via PGD Proxy, for both reads and writes, +potentially impacting performance during peak times. + +#### Adding DETACH CONCURRENTLY commands +We now offer support for DETACH CONCURRENTLY commands for EDB Postgres +Distributed (and all EDB database version types), which enables other SELECT +queries to be executed on the parent table while the DETACH operation is +underway. + +For all the Q2 EDB announcements, please visit the [EDB blog](). + diff --git a/src/pages/index.js b/src/pages/index.js index bd1bcdb01cd..b5332ae3327 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -159,8 +159,8 @@ const Page = () => { Guide - - Features + + Release Notes { headingText="Databases" to="/edb-postgres-ai/databases" > + + Databases + Cloud Service - - On Premises - - High Availability + High Availability options Date: Thu, 23 May 2024 00:42:03 +0100 Subject: [PATCH 53/67] Fix home text Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/index.mdx b/advocacy_docs/edb-postgres-ai/index.mdx index c86c0cd9a24..4023bff907f 100644 --- a/advocacy_docs/edb-postgres-ai/index.mdx +++ b/advocacy_docs/edb-postgres-ai/index.mdx @@ -15,5 +15,5 @@ navigation: - tools --- -The future home of all EDB Postgres AI documentation. Stay tuned! +The home of all EDB Postgres AI documentation. From 84412745bf44d331010c752a3e6fe5e0fdc8cbc5 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 00:54:42 +0100 Subject: [PATCH 54/67] Fix databases menu Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/databases/index.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/advocacy_docs/edb-postgres-ai/databases/index.mdx b/advocacy_docs/edb-postgres-ai/databases/index.mdx index 01e75499804..7d582172f38 100644 --- a/advocacy_docs/edb-postgres-ai/databases/index.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/index.mdx @@ -3,6 +3,10 @@ title: EDB Postgres AI Databases navTitle: Databases indexCards: simple iconName: Database +navigation: + - databases + - cloudservice + - options --- Building on decades of Postgres expertise, the EDB Postgres databases are the core of the EDB Postgres AI platform. EDB Postgres Advanced Server can take on Oracle workloads, while EDB Postgres Extended Server is designed for large-scale, mission-critical enterprise workloads. EDB Postgres Distributed provides high availability and fault tolerance for mission-critical workloads. From eb9d065e0f16995e39d9e0838f7ba272e4bad87a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 09:39:45 +0100 Subject: [PATCH 55/67] misc fixes Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx | 2 +- advocacy_docs/edb-postgres-ai/overview/guide.mdx | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx b/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx index fd190cc3da8..0eeb99399af 100644 --- a/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/cloudservice.mdx @@ -4,7 +4,7 @@ navTitle: Cloud Service description: An introduction to the EDB Postgres AI Cloud Service and its features. --- -The EDB Postgres AI Cloud Service, formerly known as BigAnimal is an evolution of the service to offer a holistic platform which offers hybrid data estate management, observability, and AI capabilities. +The EDB Postgres AI Cloud Service, formerly known as [BigAnimal](/biganimal/latest/), is an evolution of the service to offer a holistic platform which offers hybrid data estate management, observability, and AI capabilities. The EDB Postgres AI Cloud Service itself is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for AI and machine learning workloads. It also allows provides the platform for EDB Postgres AI Analytics and EDB Postgres AI Machine Learning services. diff --git a/advocacy_docs/edb-postgres-ai/overview/guide.mdx b/advocacy_docs/edb-postgres-ai/overview/guide.mdx index b4ba0f3f3bc..3dc6bcae7d8 100644 --- a/advocacy_docs/edb-postgres-ai/overview/guide.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/guide.mdx @@ -14,7 +14,7 @@ You'll want to look at the [EDB Postgres AI Machine Learning](/edb-postgres-ai/a ## Do you need to migrate your data to Postgres? -You'll want to look at the [EDB Postgres AI Migration](/edb-postgres-ai/migration) documentation, which covers the Migration Toolkit and Migration Portal. +You'll want to look at the [EDB Postgres AI Migration](/edb-postgres-ai/migration-and-ai) documentation, which covers the Migration Toolkit and Migration Portal. ## Do you want to manage your EDB Postgres AI Cloud databases? @@ -22,9 +22,9 @@ You'll want to look at the [EDB Postgres AI Console](/edb-postgres-ai/console) d ## Do you want to manage your self-managed Postgres databases as part of a hybrid data estate? -You'll want to look at the [EDB Postgres AI Platform Agent](/ebd-postgres-ai/console/agent) documentation, which covers the Estate management and on premises agent installation. +You'll want to look at the [EDB Postgres AI Platform Agent](/edb-postgres-ai/console/agent) documentation, which covers the Estate management and on premises agent installation. ## Do you want to know more about the EDB Postgres AI Cloud Service? -You'll want to look at the [EDB Postgres AI Cloud Service](/edb-postgres-ai/cloud-service) documentation, which covers the Cloud Service and its databases. +You'll want to look at the [EDB Postgres AI Cloud Service](/edb-postgres-ai/databases/cloudservice) documentation, which covers the Cloud Service and its databases. From 8918ed3e56e0891ec02c972a55f41bee0180db72 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 10:04:35 +0100 Subject: [PATCH 56/67] Link refix Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/overview/guide.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/edb-postgres-ai/overview/guide.mdx b/advocacy_docs/edb-postgres-ai/overview/guide.mdx index 3dc6bcae7d8..6a703da4b63 100644 --- a/advocacy_docs/edb-postgres-ai/overview/guide.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/guide.mdx @@ -14,7 +14,7 @@ You'll want to look at the [EDB Postgres AI Machine Learning](/edb-postgres-ai/a ## Do you need to migrate your data to Postgres? -You'll want to look at the [EDB Postgres AI Migration](/edb-postgres-ai/migration-and-ai) documentation, which covers the Migration Toolkit and Migration Portal. +You'll want to look at the [EDB Postgres AI Migration](/edb-postgres-ai/tools/migration-and-ai) documentation, which covers the Migration Toolkit and Migration Portal. ## Do you want to manage your EDB Postgres AI Cloud databases? From 7f13659b8a0fc9e48ca6f840287d63f200edff60 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 May 2024 10:23:36 +0100 Subject: [PATCH 57/67] More links and consistency fixes Signed-off-by: Dj Walker-Morgan --- .../edb-postgres-ai/databases/databases.mdx | 7 +++ .../edb-postgres-ai/databases/options.mdx | 6 +-- .../edb-postgres-ai/overview/concepts.mdx | 46 ++++++------------- src/pages/index.js | 2 +- 4 files changed, 26 insertions(+), 35 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/databases/databases.mdx b/advocacy_docs/edb-postgres-ai/databases/databases.mdx index 1d47ce64416..9a749e92ad7 100644 --- a/advocacy_docs/edb-postgres-ai/databases/databases.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/databases.mdx @@ -10,11 +10,18 @@ EDB Postgres databases are the core of the EDB Postgres AI platform. EDB Postgre EDB Postgres Advanced Server (EPAS) is an enhanced version of PostgreSQL that is designed to meet the needs of large-scale, mission-critical enterprise workloads. EPAS is built on the open source PostgreSQL database, and includes additional enterprise-class features and capabilities that are critical for enterprise database deployments. These include Oracle compatibility and transparent data encryption. EPAS is available for self-managed deployment and on the EDB Postgres AI Cloud Service. +* Read more about [EDB Postgres Advanced Server](/epas/latest/). + ## EDB Postgres Extended Server (PGE) EDB Postgres Extended Server (PGE) is an enhanced version of PostgreSQL that is designed to meet the needs of large-scale, mission-critical enterprise workloads. PGE is built on the open source PostgreSQL database, and includes additional enterprise-class features and capabilities that are critical for enterprise database deployments. This includes transparent data encryption. PGE is available for self-managed deployment and on the EDB Postgres AI Cloud Service. +* Read more about [EDB Postgres Extended Server](/pge/latest/). + ## EDB Postgres Distributed EDB Postgres Distributed (PGD) is a high availability solution for EDB Postgres databases. PGD provides a distributed database environment that is designed to ensure high availability and fault tolerance for mission-critical workloads. PGD can be used with EPAS, PGE or PostgreSQL databases. PGD is available for self-managed deployment and on the EDB Postgres AI Cloud Service (as the Distributed High Availability option). +* Read more about [EDB Postgres Distributed](/pgd/latest/). + + diff --git a/advocacy_docs/edb-postgres-ai/databases/options.mdx b/advocacy_docs/edb-postgres-ai/databases/options.mdx index cd28fbf38f9..4fd95b0ebdf 100644 --- a/advocacy_docs/edb-postgres-ai/databases/options.mdx +++ b/advocacy_docs/edb-postgres-ai/databases/options.mdx @@ -1,11 +1,11 @@ --- title: EDB Postgres AI Databases - Deployment Options navTitle: Deployment Options -description: High Availability and other available for EDB Postgres AI Databases and on EDB Postgres AI Cloud Service. +description: High Availability and other options available for EDB Postgres AI Databases and on EDB Postgres AI Cloud Service. deepToC: true --- -## Availablitity Options +## Availability Options ### Single Instance @@ -19,7 +19,7 @@ This configuration provides fault tolerance and high availability in a particula This is a standard configuration option on EDB Postgres AI Cloud Service. -## Distributed High Availability +### Distributed High Availability High availability is a critical requirement for mission-critical workloads. EDB Postgres Distributed (PGD) provides a distributed database environment that is designed to ensure high availability and fault tolerance for mission-critical workloads. PGD can be used with EPAS, PGE or PostgreSQL databases. PGD is available for self-managed deployment and on the EDB Postgres AI Cloud Service (as the Distributed High Availability option). diff --git a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx index a043d683c24..96c9b3ffb40 100644 --- a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx @@ -5,50 +5,34 @@ deepToc: true description: A look at the concepts that underpin EDB Postgres AI. --- -## What is EDB Postgres AI? - EDB Postgres AI takes EDB’s leading expertise in Postgres and expands the scope of Postgres to address modern challenges. From simplifying your database estate management to infusing AI deep into Postgres and putting it to work to bring all your data under one analytical eye. EDB Postgres AI is composed of multiple elements which come together to deliver a unified and powerful experience: -### EDB Postgres AI - Console: - +## [EDB Postgres AI - Console](/edb-postgres-ai/console): * Providing the single pane of glass onto all your EDB Postgres AI operations, Console is there to manage your database landscape. - * #### Integrated management: + * ### [Hybrid estate management](/edb-postgres-ai/console/estate): * EDB Postgres AI Cloud databases are automatically managed on the Console. - * #### EDB Postgres AI Agent: + * ### [EDB Postgres AI Agent](/edb-postgres-ai/console/agent): * On premises databases can be brought under one manageable Console view with the Agent enabling an unprecedented view of diverse deployments. - -### EDB Postgres AI - Databases: - +## [EDB Postgres AI - Databases](/edb-postgres-ai/databases): * All of EDB’s database expertise can be found in EDB Postgres Advanced Server and EDB Postgres Extended Server. * Oracle compatibility, transparent data encryption and more. They provide the data fabric on which EDB Postgres AI operates. * Combined with EDB Postgres Distributed, they can also provide a high availability environment for your data. * All of these components are available on the EDB Postgres AI Cloud Service, and managed through the EDB Postgres AI Console. - * #### EDB Postgres Advanced Server and EDB Postgres Extended Server: - * … - * #### EDB Postgres Distributed: - * … - * #### EDB Postgres AI Cloud Service: + * ### [EDB Postgres Advanced Server and EDB Postgres Extended Server](/edb-postgres-ai/databases/databases): + * EDB Postgres Advanced Server and EDB Postgres Extended Server both provide mission critical capabilities for your data, with EDB Postgres Advanced Server also providing Oracle compatibility. + * ### [EDB Postgres Distributed](/edb-postgres-ai/databases/options/): + * High availability with an active-active mesh of Postgres instances, EDB Postgres Distributed provides a robust and scalable environment for your data. + * ### [EDB Postgres AI Cloud Service](/edb-postgres-ai/databases/cloudservice): * Not just databases, but driven by databases, Cloud Service provides a global platform for delivering new elements of EDB Postgres AI efficiently and effectively. - -### EDB Postgres AI - Analytics: - -* Filtering out the data noise and revealing insights and value, Analytics brings both structured relational data in Postgres and unstructured data in object storage together for exploration. At the heart of Analytics is a custom built store for this data: - -### EDB Postgres AI - Lakehouse: - -* Built to bring structured and unstructured data together, Lakehouse supports numerous formats to bring your data in from the cold, ready to be analyzed. - -### EDB Postgres AI - AI/ML: - -* Postgres has proven its capability as a flexible data environment, and Vector data, the core of generative AI, is already infused into EDB Postgres AI providing a platform for a range of practical and effective AI/ML solutions. - -### EDB Postgres AI - Platforms and Tools: - +## [EDB Postgres AI - Lakehouse Analytics](/edb-postgres-ai/analytics): +* Filtering out the data noise and revealing insights and value, Lakehouse Analytics brings both structured relational data in Postgres and unstructured data in object storage together for exploration. At the heart of Analytics is a custom built store for this data: +* Built to bring structured and unstructured data together, Lakehouse Nodes support numerous formats to bring your data in from the cold, ready to be analyzed. +## [EDB Postgres AI - AI/ML](/edb-postgres-ai/ai-ml): +* Postgres has proven its capability as a flexible data environment, and Vector data, the core of generative AI, is already infused into EDB Postgres AI providing a platform for a range of practical and effective AI/ML solutions. A technical preview of this capability is available for the pgai extension. +## [EDB Postgres AI - Platforms and Tools](/edb-postgres-ai/tools): * Postgres’s extensions are a source of its power and popularity, and are one of the categories that fall within this element of EDB Postgres AI. - * Extensions sit alongside existing applications like Postgres Enterprise Manager, Barman, and Query Advisor as tools that allow you to leverage Postgres’s capabilities. - * Also within this element are EDB’s Migration tools, Migration Toolkit and Migration Portal. The Migration Portal is among the first EDB tools to include embedded AI with an AI copilot that can assist users in developing migration strategies. diff --git a/src/pages/index.js b/src/pages/index.js index b5332ae3327..b5b48f0d147 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -190,7 +190,7 @@ const Page = () => { Cloud Service - High Availability options + Deployment options Date: Thu, 23 May 2024 10:43:50 +0100 Subject: [PATCH 58/67] Title tidy and remove errant deeptoc Signed-off-by: Dj Walker-Morgan --- advocacy_docs/edb-postgres-ai/overview/concepts.mdx | 1 - advocacy_docs/edb-postgres-ai/tools/index.mdx | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx index 96c9b3ffb40..c8665eb4d9e 100644 --- a/advocacy_docs/edb-postgres-ai/overview/concepts.mdx +++ b/advocacy_docs/edb-postgres-ai/overview/concepts.mdx @@ -1,7 +1,6 @@ --- title: EDB Postgres AI Overview - Concepts navTitle: Concepts -deepToc: true description: A look at the concepts that underpin EDB Postgres AI. --- diff --git a/advocacy_docs/edb-postgres-ai/tools/index.mdx b/advocacy_docs/edb-postgres-ai/tools/index.mdx index f93637e4d74..ebd6205c435 100644 --- a/advocacy_docs/edb-postgres-ai/tools/index.mdx +++ b/advocacy_docs/edb-postgres-ai/tools/index.mdx @@ -9,7 +9,7 @@ navigation: - backup --- -Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. +Tools - Everything you need to manage your EDB Postgres AI databases, from migration to backup and recovery. EDB Postgres AI Tools is a set of tools, utilities and extensions that are designed to help you manage your EDB Postgres AI databases. From migration to backup and recovery, EDB Postgres AI Tools has you covered. From 5b8dd42d19beaa8128598d2217935c252c413d0d Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Tue, 21 May 2024 01:05:46 +0100 Subject: [PATCH 59/67] Separate Storage from Compute for reduced time to insights --- advocacy_docs/edb-postgres-ai/analytics/analytics.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx b/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx index 9e10591e39b..8f7d6ff8f9c 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx @@ -7,3 +7,7 @@ description: Discover how EDB Postgres AI Analytics can help you turn data into An Introduction to EDB Postgres AI Analytics. _Corrall your data and then deep dive into it with EDB Postgres AI Analytics. This section provides an overview of the tools and features that help you to gather, analyze, and visualize data. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ + +# Relax by the Lakehouse + +Rest assured your compute and storage will remain separate. \ No newline at end of file From 18b9cfee0e16a5d0b86ad6bfe5346f274192f16b Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Wed, 22 May 2024 01:04:28 +0100 Subject: [PATCH 60/67] Add stub and some starting docs for all PG lakehouse pages --- .../analytics/benchmarking_notesbooks.mdx | 9 ++++ .../analytics/connect_to_lakehouse_node.mdx | 43 ++++++++++++++++ .../analytics/delta_tables_reference.mdx | 11 ++++ .../analytics/external_buckets.mdx | 32 ++++++++++++ .../edb-postgres-ai/analytics/getstarted.mdx | 10 ---- .../edb-postgres-ai/analytics/index.mdx | 31 ++++++++++-- .../{lakehouse.mdx => lakehouse_nodes.mdx} | 7 +-- .../analytics/lakehouse_sdl_tool.mdx | 7 +++ .../analytics/lakehouse_sync.mdx | 9 ++++ .../analytics/lakehouse_tables.mdx | 13 +++++ .../analytics/launch_lakehouse_node.mdx | 7 +++ .../analytics/{analytics.mdx => overview.mdx} | 4 +- .../analytics/platform_compatibility.mdx | 48 ++++++++++++++++++ .../analytics/pricing_reference.mdx | 14 ++++++ .../analytics/query_sample_data.mdx | 45 +++++++++++++++++ .../edb-postgres-ai/analytics/quick_start.mdx | 8 +++ .../edb-postgres-ai/analytics/roadmap.mdx | 16 ++++++ .../analytics/sample_datasets.mdx | 24 +++++++++ .../analytics/storage_locations.mdx | 8 +++ .../analytics/sync_to_lakehouse.mdx | 50 +++++++++++++++++++ .../edb-postgres-ai/analytics/terminology.mdx | 19 +++++++ .../analytics/user_management.mdx | 23 +++++++++ .../what_to_expect_when_querying.mdx | 7 +++ 23 files changed, 427 insertions(+), 18 deletions(-) create mode 100644 advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx rename advocacy_docs/edb-postgres-ai/analytics/{lakehouse.mdx => lakehouse_nodes.mdx} (61%) create mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx rename advocacy_docs/edb-postgres-ai/analytics/{analytics.mdx => overview.mdx} (88%) create mode 100644 advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/terminology.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/user_management.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx diff --git a/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx b/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx new file mode 100644 index 00000000000..991f514dc2a --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx @@ -0,0 +1,9 @@ +--- +title: Benchmarking Notebooks - EDB Postgres AI Analytics +navTitle: Benchmarking Notebooks +description: We have some Jupyter notebooks that are ready-to-run against the benchmarking datasets available to every Postgres node. +--- + +To assist you in getting started, we have a GitHub repository containing Jupyter notebooks that you can use for benchmarking Lakehouse Nodes, using the benchmark data that comes pre-loaded with them. + +You can get them here... \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx b/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx new file mode 100644 index 00000000000..1c65d8b44b8 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx @@ -0,0 +1,43 @@ +--- +title: Connect to Lakehouse Node - EDB Postgres AI Analytics +navTitle: Connect to Lakehouse Node +description: Lakehouse Nodes are "just Postgres," so you can connect to them with any Postgres client. +--- + +To connect to Lakehouse Node, get the connection string + +## Getting the Connection String + +You can find this in the "clusters" page of Postgres AI Console. + +## Example: pgcli + +Here's an example of connecting with pgcli. + +## Example: psql + +Here's an example of connecting with psql. + +## Using .pgpass file + +You don't need to enter your password every time. + +## Gotchas and Limitations + +In general, introspection queries can be "weird" + +## Tested and Working Clients + +These clients are known tested and working. + +* psql +* pgcli +* pgadmin +* psycopg2 +* Tableau +* Metabase +* Superset + +## Clients with known issues + +None at this time. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx new file mode 100644 index 00000000000..5622d8e4fb1 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx @@ -0,0 +1,11 @@ +--- +title: Delta Tables Reference - EDB Postgres AI Analytics +navTitle: Delta Tables +description: Lakehouse Tables are Delta Tables that follow the Delta Lake protocol. +--- + +Lakehouse Tables use an open format called Delta Lake protocol. A "Delta Table" is a well-defined spec for columnar tables. Basically, it's a "folder" containing a set of Parquet files and a JSON file with metadata about those files. + +Many tools can create Delta tables. + +blahblah \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx b/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx new file mode 100644 index 00000000000..358a0c6e55f --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx @@ -0,0 +1,32 @@ +--- +title: External Buckets - EDB Postgres AI Analytics +navTitle: External Buckets +description: Learn how to query Delta Tables in external buckets. +--- + +It's possible to query data in external buckets, assuming it's formatted correctly as Delta Tables and meets a few requirements. + +This is an early feature, and there is one major caveat: this must be a public bucket. + +We're working on support for private buckets, and will be shipping it shortly. In fact, we actually support it now, but it requires some manual action on our side, and your side, for managing IAM policies. We want to get this right without managing credentials. + +## Requirements + +These are the requirements + +* Must be an S3 bucket +* Must be _publicly accessible_ +* Must be formatted as Delta Tables in `schema/table` format, where `schema` and `table` are valid Postgres identifiers + +## Loading Data + +You can use `lakehouse-sdl` to load data into a storage location... + +## Pointing to a Bucket + +You can call `seafowl.set_bucket_location(...)` + +## Resetting Bucket Location + +You can call `seafowl.set_bucket_location(null)` + diff --git a/advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx b/advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx deleted file mode 100644 index 9fc3a50c48c..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/getstarted.mdx +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: EDB Postgres AI Analytics - Get Started -navTitle: Get Started -description: Where to begin with EDB Postgres AI Analytics and how to get results quickly. ---- - -Where to begin with EDB Postgres AI Analytics and how to get results quickly. - -_The EDB Postgres AI Analytics is a fully managed cloud service that provides a high-performance, scalable, and secure database platform for analytics workloads. The service is built on the EDB Postgres Advanced Server database and is designed to help organizations accelerate the development and deployment of analytics applications._ - diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 2a0a257aab6..6227902939e 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -4,9 +4,34 @@ navTitle: Analytics indexCards: simple iconName: Improve navigation: -- getstarted -- analytics -- lakehouse +- "#Concepts" +- terminology +- overview +- lakehouse_nodes +- lakehouse_tables +- storage_locations +- lakehouse_sync +- "#Get Started" +- quick_start +- "#Operating" +- launch_lakehouse_node +- user_management +- "#Querying Data" +- connect_to_lakehouse_node +- query_sample_data +- what_to_expect_when_querying +- "#Loading Data" +- sync_to_lakehouse +- external_buckets +- "#Reference" +- platform_compatibility +- sample_datasets +- delta_tables_reference +- pricing_reference +- roadmap +- "#Tools" +- lakehouse_sdl_tool +- benchmarking_notebooks --- Introduction to EDB Postgres AI Analytics. diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx similarity index 61% rename from advocacy_docs/edb-postgres-ai/analytics/lakehouse.mdx rename to advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx index ed27d019715..694bbe6a63d 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/lakehouse.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx @@ -1,9 +1,10 @@ --- -title: EDB Postgres AI Databases- Lakehouse -navTitle: Lakehouse -description: The EDB Postgres AI Lakehouse provides a single source of truth for all your data. +title: Lakehouse Nodes - EDB Postgres AI Analytics +navTitle: Lakehouse Nodes +description: A Lakehouse Node is Postgres with a vectorized query engine. Lakehouse Nodes query Lakehouse Tables in Storage Locations. --- + _Bring all your data together with the EDB Postgres AI Lakehouse. Get a single source of truth for your Analytics needs. The Lakehouse is a unified data platform that combines the best of data lakes and data warehouses. With a single source of truth for all your data, you are able to analyze and visualize data in real time_ diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx new file mode 100644 index 00000000000..4682a3634c8 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx @@ -0,0 +1,7 @@ +--- +title: lakehouse_sdl Tool - EDB Postgres AI Analytics +navTitle: lakehouse_sdl +description: lakehouse_sdl is a simple tool for creating Lakehouse tables from the result of arbitrary Postgres queries. +--- + +You can get it here... \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx new file mode 100644 index 00000000000..c62b4ff2b49 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx @@ -0,0 +1,9 @@ +--- +title: Lakehouse Sync - EDB Postgres AI Analytics +navTitle: Lakehouse Sync +description: Lakehouse Sync creates Lakehouse Tables in Storage Locations +--- + +Lakehouse Sync is how you load data into the Lakehouse. + +Lakehouse Sync creates Lakehouse Tables in Storage Locations. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx new file mode 100644 index 00000000000..82ec6c35fdf --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx @@ -0,0 +1,13 @@ +--- +title: Lakehouse Tables - EDB Postgres AI Analytics +navTitle: Lakehouse Tables +description: Lakehouse Tables are columnar, open table formats queryable with Lakehouse Nodes. +--- + +_Lakehouse Tables are at the core of EDB Postgres AI Analytics workloads._ + + +# Delta Tables + +We use Delta Tables + diff --git a/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx b/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx new file mode 100644 index 00000000000..3418fc5c0ff --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx @@ -0,0 +1,7 @@ +--- +title: Launch Lakehouse Node - EDB Postgres AI Analytics +navTitle: Launch Lakehouse Node +description: Provision a Lakehouse Node from the EDB Postgres AI Console. +--- + +Launch a Lakehouse Node from the EDB Postgres AI Console. diff --git a/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx b/advocacy_docs/edb-postgres-ai/analytics/overview.mdx similarity index 88% rename from advocacy_docs/edb-postgres-ai/analytics/analytics.mdx rename to advocacy_docs/edb-postgres-ai/analytics/overview.mdx index 8f7d6ff8f9c..7ba5bd7b472 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/analytics.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/overview.mdx @@ -1,6 +1,6 @@ --- -title: EDB Postgres AI Analytics - Analytics -navTitle: Analytics +title: Overview - EDB Postgres AI Analytics +navTitle: Overview description: Discover how EDB Postgres AI Analytics can help you turn data into insights. --- diff --git a/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx b/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx new file mode 100644 index 00000000000..fb714d0c1e8 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx @@ -0,0 +1,48 @@ +--- +title: Platform Compatiblility - EDB Postgres AI Analytics +navTitle: Platform Compatiblility +description: Where can you run EDB Postgres AI Analytics? +--- + +Currently, EDB Postgres AI Analytics is a Cloud solution, and generally speaking it's limited to AWS. Support for GCP and Azure is forthcoming, and eventually so is on-premises support. + +# Support + +## AWS + +* AWS, in EDB Hosted Regions + +Supported instance types: + +* NVMe ... + +## Soon + +* AWS, in "Bring Your Own Account" Regions + +## Future + +* GCP +* Azure +* On-premises + +# Lakehouse Sync Interoperability + +## Sources + +* AWS, EDB Cloud Hosted Databases + +## Destinations + +* Managed Storage Locations in the same account as the source database (i.e., EDB Hosted) + +## Coming Soon + +* Bring Your Own Account +* Cross-account (e.g., sync from BYOA to EDB Hosted) + +## Coming Eventually + +* Non-Postgres sources +* Postgres sources in other clouds +* On-premises and unmanaged Postgres sources diff --git a/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx new file mode 100644 index 00000000000..269d374916f --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx @@ -0,0 +1,14 @@ +--- +title: Pricing Reference - EDB Postgres AI Analytics +navTitle: Pricing +description: Lakehouse Nodes are billed on a consumption basis using EPAS and PGE Licenses. +--- + +Lakehouse Nodes are billed per hour and have two components: + +* Infrastructure charges +* License charges + +Infrastructure is passed through from costs of the cloud provider. + +License costs are EPAS or PGE. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx b/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx new file mode 100644 index 00000000000..1f7f283351f --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx @@ -0,0 +1,45 @@ +--- +title: Query Sample Data - EDB Postgres AI Analytics +navTitle: Query Sample Data +description: Every Lakehouse Node comes pre-configured to query sample datasets in a nearby Storage Location. +--- + +We have some datasets pre-loaded into a storage bucket in a nearby region. + +These buckets are public. + +## Inspecting Available Datasets + +Just run `\dn` + +```sql +\dn +``` + +## TPC-H + +TPC-H datasets are available in the following scale factors: + +* Scale Factor 1 +* Scale Factor 10 +* Scale Factor 100 +* Scale Factor 1000 + +## TPC-DS + +TPC-DS datasets are available in the following scale factors: + +* Scale Factor 1 +* Scale Factor 10 +* Scale Factor 100 +* Scale Factor 1000 + +## Clickbench + +## Billion Row Challenge + +## Default Bucket Names + +## Note on HammerDB and other tools for loading datasets + +These datasets are pre-provisioned. It's not possible to write to the nodes. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx new file mode 100644 index 00000000000..404fedac75d --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx @@ -0,0 +1,8 @@ +--- +title: Quick Start - EDB Postgres AI Analytics +navTitle: Quick Start +description: Launch a Lakehouse Node and query sample data. +--- + +Get started by querying sample data. + diff --git a/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx b/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx new file mode 100644 index 00000000000..22dda70a721 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx @@ -0,0 +1,16 @@ +--- +title: Roadmap - EDB Postgres AI Analytics +navTitle: Roadmap +description: Where are we going with all this? +--- + +This is an early product, and we want you to get excited about the performance and possibilities. However, there's still a lot of work to do, especially as it relates to laoding data... + +Here are some of the things we're planning to support: + +* Continuous replication +* On-premises deployment +* Private buckets +* "Bring Your Own Bucket" (not just set in each node) +* Catalogs + diff --git a/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx b/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx new file mode 100644 index 00000000000..d330a4d53e2 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx @@ -0,0 +1,24 @@ +--- +title: Sample Datasets - EDB Postgres AI Analytics +navTitle: Sample Datasets +description: What datasets are available by default to Lakehouse Nodes? +--- + +When you launch a Lakehouse Node, it can query pre-loaded sample datasets in a public bucket in the same region. + +These are the datasets: + +- TPC-H SF 1 +- TPC-H SF 10 +- TPC-H SF 100 +- TPC-H SF 1000 +- TPC-DS SF 1 +- TPC-DS SF 10 +- TPC-DS SF 100 +- TPC-DS SF 1000 +- Clickbench + +These are the default bucket names: + +- blahblah + diff --git a/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx b/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx new file mode 100644 index 00000000000..355b1a336b3 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx @@ -0,0 +1,8 @@ +--- +title: Storage Locations - EDB Postgres AI Analytics +navTitle: Storage Locations +description: Storage Locations are locations in object storage containing Lakehouse Tables. +--- + +Storage Locations are locations in object storage where you store Lakehouse Tables. + diff --git a/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx b/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx new file mode 100644 index 00000000000..812497ff61e --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx @@ -0,0 +1,50 @@ +--- +title: Sync to Lakehouse - EDB Postgres AI Analytics +navTitle: Sync to Lakehouse +description: Learn how to sync data from a transactional database to Lakehouse Tables in object storage. +--- + +This guide will teach you how to sync data from a transactional database to Lakehouse Tables in a Managed Storage Location. + +## Assumptions and Goal + +We have some tables in a transactional database, and we want to sync them to Lakehouse Tables in a Managed Storage Location. + +Caveats, source database: + +* Must be a Postgres cluster in EDB Postgres AI Cloud Database +* Must be a managed Postgres AI Cloud Database +* Must be in AWS +* Must be in an EDB Hosted AWS region (not "Bring Your Own Account") + +Caveats, managed storage location: + +* Must be in the same project +* Must be in the same EDB Hosted AWS region +* Must be a _Managed_ Storage location - support for "bring your own bucket" destinations is coming soon... +* You cannot delete data in the Managed Storage Location + +Limitations: + +* One time sync +* Updating data requires performing another Sync to Lakehouse + +## Setup + +Create a transactional database... single node okay for this demo... + +## Create the migration + +1. Go to the cluster +2. Create the migration +3. Wait for it to complete + +## Start a Lakehouse Node + +1. Start Lakehouse Node +2. Query tables as defined in the migration + +## Start another Lakehouse Node + +Any Lakehouse Node in the project can query any Lakehouse Tables in the project. + diff --git a/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx b/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx new file mode 100644 index 00000000000..7b7bf2b197b --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx @@ -0,0 +1,19 @@ +--- +title: Terminology - EDB Postgres AI Analytics +navTitle: Terminology +description: The Lakehouse lexicon and terms to learn about analytics on EDB Postgres AI. +--- + +This terminology list includes terms associated with Analytics on EDB Postgres AI. + +### Lakehouse + +Lakehouse + +### Storage Location + +An organizational concept... + +#### Managed Storage Location + +A storage location where we manage the layout for you \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx b/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx new file mode 100644 index 00000000000..d198c3c15dc --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx @@ -0,0 +1,23 @@ +--- +title: User Management - EDB Postgres AI Analytics +navTitle: User Management +description: Manage access to Lakehouse Nodes +--- + +For access to Postgres AI Console, see + +Otherwise, access to Lakehouse Nodes is governed by the same rules as Postgres user management. + +## edb_admin + +When provisioning a cluster, a user `edb_admin` is created... + +This is the default + +### Creating users + +You can use Postgres user management to create users + +### Gotchas and Limitations + +When tables are re-created, grants will be lost... diff --git a/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx b/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx new file mode 100644 index 00000000000..1c1fb722e41 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx @@ -0,0 +1,7 @@ +--- +title: What to Expect When Querying - EDB Postgres AI Analytics +navTitle: What to Expect +description: In general, queries are "read only." All SELECT statements will work, but you cannot manipulate tables directly (at least not yet). +--- + +Here's what to expect.... \ No newline at end of file From ebd4007bed741e27f9a47e431dc367450ea0749e Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Wed, 22 May 2024 22:32:51 +0100 Subject: [PATCH 61/67] Remove ambitious content outline in favor of three items Content remains TBD, but now only three pages. --- .../analytics/benchmarking_notesbooks.mdx | 9 ---- .../edb-postgres-ai/analytics/concepts.mdx | 7 +++ .../analytics/connect_to_lakehouse_node.mdx | 43 ---------------- .../analytics/delta_tables_reference.mdx | 11 ---- .../analytics/external_buckets.mdx | 32 ------------ .../edb-postgres-ai/analytics/index.mdx | 29 +---------- .../analytics/lakehouse_nodes.mdx | 10 ---- .../analytics/lakehouse_sdl_tool.mdx | 7 --- .../analytics/lakehouse_sync.mdx | 9 ---- .../analytics/lakehouse_tables.mdx | 13 ----- .../analytics/launch_lakehouse_node.mdx | 7 --- .../edb-postgres-ai/analytics/overview.mdx | 13 ----- .../analytics/platform_compatibility.mdx | 48 ------------------ .../analytics/pricing_reference.mdx | 14 ------ .../analytics/query_sample_data.mdx | 45 ----------------- .../edb-postgres-ai/analytics/quick_start.mdx | 2 +- .../edb-postgres-ai/analytics/reference.mdx | 8 +++ .../edb-postgres-ai/analytics/roadmap.mdx | 16 ------ .../analytics/sample_datasets.mdx | 24 --------- .../analytics/storage_locations.mdx | 8 --- .../analytics/sync_to_lakehouse.mdx | 50 ------------------- .../edb-postgres-ai/analytics/terminology.mdx | 19 ------- .../analytics/user_management.mdx | 23 --------- .../what_to_expect_when_querying.mdx | 7 --- 24 files changed, 18 insertions(+), 436 deletions(-) delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/concepts.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/overview.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx create mode 100644 advocacy_docs/edb-postgres-ai/analytics/reference.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/terminology.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/user_management.mdx delete mode 100644 advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx diff --git a/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx b/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx deleted file mode 100644 index 991f514dc2a..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/benchmarking_notesbooks.mdx +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Benchmarking Notebooks - EDB Postgres AI Analytics -navTitle: Benchmarking Notebooks -description: We have some Jupyter notebooks that are ready-to-run against the benchmarking datasets available to every Postgres node. ---- - -To assist you in getting started, we have a GitHub repository containing Jupyter notebooks that you can use for benchmarking Lakehouse Nodes, using the benchmark data that comes pre-loaded with them. - -You can get them here... \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx b/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx new file mode 100644 index 00000000000..3380f3e31d4 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx @@ -0,0 +1,7 @@ +--- +title: Concepts - EDB Postgres Lakehouse +navTitle: Concepts +description: Learn about the ideas and terminology behind EDB Postgres Lakehouse for Analytics workloads. +--- + +Concepts about stuff. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx b/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx deleted file mode 100644 index 1c65d8b44b8..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/connect_to_lakehouse_node.mdx +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Connect to Lakehouse Node - EDB Postgres AI Analytics -navTitle: Connect to Lakehouse Node -description: Lakehouse Nodes are "just Postgres," so you can connect to them with any Postgres client. ---- - -To connect to Lakehouse Node, get the connection string - -## Getting the Connection String - -You can find this in the "clusters" page of Postgres AI Console. - -## Example: pgcli - -Here's an example of connecting with pgcli. - -## Example: psql - -Here's an example of connecting with psql. - -## Using .pgpass file - -You don't need to enter your password every time. - -## Gotchas and Limitations - -In general, introspection queries can be "weird" - -## Tested and Working Clients - -These clients are known tested and working. - -* psql -* pgcli -* pgadmin -* psycopg2 -* Tableau -* Metabase -* Superset - -## Clients with known issues - -None at this time. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx deleted file mode 100644 index 5622d8e4fb1..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/delta_tables_reference.mdx +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: Delta Tables Reference - EDB Postgres AI Analytics -navTitle: Delta Tables -description: Lakehouse Tables are Delta Tables that follow the Delta Lake protocol. ---- - -Lakehouse Tables use an open format called Delta Lake protocol. A "Delta Table" is a well-defined spec for columnar tables. Basically, it's a "folder" containing a set of Parquet files and a JSON file with metadata about those files. - -Many tools can create Delta tables. - -blahblah \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx b/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx deleted file mode 100644 index 358a0c6e55f..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/external_buckets.mdx +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: External Buckets - EDB Postgres AI Analytics -navTitle: External Buckets -description: Learn how to query Delta Tables in external buckets. ---- - -It's possible to query data in external buckets, assuming it's formatted correctly as Delta Tables and meets a few requirements. - -This is an early feature, and there is one major caveat: this must be a public bucket. - -We're working on support for private buckets, and will be shipping it shortly. In fact, we actually support it now, but it requires some manual action on our side, and your side, for managing IAM policies. We want to get this right without managing credentials. - -## Requirements - -These are the requirements - -* Must be an S3 bucket -* Must be _publicly accessible_ -* Must be formatted as Delta Tables in `schema/table` format, where `schema` and `table` are valid Postgres identifiers - -## Loading Data - -You can use `lakehouse-sdl` to load data into a storage location... - -## Pointing to a Bucket - -You can call `seafowl.set_bucket_location(...)` - -## Resetting Bucket Location - -You can call `seafowl.set_bucket_location(null)` - diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 6227902939e..53ef24a8470 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -4,34 +4,9 @@ navTitle: Analytics indexCards: simple iconName: Improve navigation: -- "#Concepts" -- terminology -- overview -- lakehouse_nodes -- lakehouse_tables -- storage_locations -- lakehouse_sync -- "#Get Started" +- concepts - quick_start -- "#Operating" -- launch_lakehouse_node -- user_management -- "#Querying Data" -- connect_to_lakehouse_node -- query_sample_data -- what_to_expect_when_querying -- "#Loading Data" -- sync_to_lakehouse -- external_buckets -- "#Reference" -- platform_compatibility -- sample_datasets -- delta_tables_reference -- pricing_reference -- roadmap -- "#Tools" -- lakehouse_sdl_tool -- benchmarking_notebooks +- reference --- Introduction to EDB Postgres AI Analytics. diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx deleted file mode 100644 index 694bbe6a63d..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_nodes.mdx +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: Lakehouse Nodes - EDB Postgres AI Analytics -navTitle: Lakehouse Nodes -description: A Lakehouse Node is Postgres with a vectorized query engine. Lakehouse Nodes query Lakehouse Tables in Storage Locations. ---- - - -_Bring all your data together with the EDB Postgres AI Lakehouse. Get a single source of truth for your Analytics needs. The Lakehouse is a unified data platform that combines the best of data lakes and data warehouses. With a single source of truth for all your data, you are able to analyze and visualize data in real time_ - - diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx deleted file mode 100644 index 4682a3634c8..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sdl_tool.mdx +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: lakehouse_sdl Tool - EDB Postgres AI Analytics -navTitle: lakehouse_sdl -description: lakehouse_sdl is a simple tool for creating Lakehouse tables from the result of arbitrary Postgres queries. ---- - -You can get it here... \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx deleted file mode 100644 index c62b4ff2b49..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_sync.mdx +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Lakehouse Sync - EDB Postgres AI Analytics -navTitle: Lakehouse Sync -description: Lakehouse Sync creates Lakehouse Tables in Storage Locations ---- - -Lakehouse Sync is how you load data into the Lakehouse. - -Lakehouse Sync creates Lakehouse Tables in Storage Locations. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx b/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx deleted file mode 100644 index 82ec6c35fdf..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/lakehouse_tables.mdx +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: Lakehouse Tables - EDB Postgres AI Analytics -navTitle: Lakehouse Tables -description: Lakehouse Tables are columnar, open table formats queryable with Lakehouse Nodes. ---- - -_Lakehouse Tables are at the core of EDB Postgres AI Analytics workloads._ - - -# Delta Tables - -We use Delta Tables - diff --git a/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx b/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx deleted file mode 100644 index 3418fc5c0ff..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/launch_lakehouse_node.mdx +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Launch Lakehouse Node - EDB Postgres AI Analytics -navTitle: Launch Lakehouse Node -description: Provision a Lakehouse Node from the EDB Postgres AI Console. ---- - -Launch a Lakehouse Node from the EDB Postgres AI Console. diff --git a/advocacy_docs/edb-postgres-ai/analytics/overview.mdx b/advocacy_docs/edb-postgres-ai/analytics/overview.mdx deleted file mode 100644 index 7ba5bd7b472..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/overview.mdx +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: Overview - EDB Postgres AI Analytics -navTitle: Overview -description: Discover how EDB Postgres AI Analytics can help you turn data into insights. ---- - -An Introduction to EDB Postgres AI Analytics. - -_Corrall your data and then deep dive into it with EDB Postgres AI Analytics. This section provides an overview of the tools and features that help you to gather, analyze, and visualize data. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ - -# Relax by the Lakehouse - -Rest assured your compute and storage will remain separate. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx b/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx deleted file mode 100644 index fb714d0c1e8..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/platform_compatibility.mdx +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: Platform Compatiblility - EDB Postgres AI Analytics -navTitle: Platform Compatiblility -description: Where can you run EDB Postgres AI Analytics? ---- - -Currently, EDB Postgres AI Analytics is a Cloud solution, and generally speaking it's limited to AWS. Support for GCP and Azure is forthcoming, and eventually so is on-premises support. - -# Support - -## AWS - -* AWS, in EDB Hosted Regions - -Supported instance types: - -* NVMe ... - -## Soon - -* AWS, in "Bring Your Own Account" Regions - -## Future - -* GCP -* Azure -* On-premises - -# Lakehouse Sync Interoperability - -## Sources - -* AWS, EDB Cloud Hosted Databases - -## Destinations - -* Managed Storage Locations in the same account as the source database (i.e., EDB Hosted) - -## Coming Soon - -* Bring Your Own Account -* Cross-account (e.g., sync from BYOA to EDB Hosted) - -## Coming Eventually - -* Non-Postgres sources -* Postgres sources in other clouds -* On-premises and unmanaged Postgres sources diff --git a/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx deleted file mode 100644 index 269d374916f..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/pricing_reference.mdx +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Pricing Reference - EDB Postgres AI Analytics -navTitle: Pricing -description: Lakehouse Nodes are billed on a consumption basis using EPAS and PGE Licenses. ---- - -Lakehouse Nodes are billed per hour and have two components: - -* Infrastructure charges -* License charges - -Infrastructure is passed through from costs of the cloud provider. - -License costs are EPAS or PGE. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx b/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx deleted file mode 100644 index 1f7f283351f..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/query_sample_data.mdx +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: Query Sample Data - EDB Postgres AI Analytics -navTitle: Query Sample Data -description: Every Lakehouse Node comes pre-configured to query sample datasets in a nearby Storage Location. ---- - -We have some datasets pre-loaded into a storage bucket in a nearby region. - -These buckets are public. - -## Inspecting Available Datasets - -Just run `\dn` - -```sql -\dn -``` - -## TPC-H - -TPC-H datasets are available in the following scale factors: - -* Scale Factor 1 -* Scale Factor 10 -* Scale Factor 100 -* Scale Factor 1000 - -## TPC-DS - -TPC-DS datasets are available in the following scale factors: - -* Scale Factor 1 -* Scale Factor 10 -* Scale Factor 100 -* Scale Factor 1000 - -## Clickbench - -## Billion Row Challenge - -## Default Bucket Names - -## Note on HammerDB and other tools for loading datasets - -These datasets are pre-provisioned. It's not possible to write to the nodes. \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx index 404fedac75d..8831edabd21 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx @@ -1,5 +1,5 @@ --- -title: Quick Start - EDB Postgres AI Analytics +title: Quick Start - EDB Postgres Lakehouse navTitle: Quick Start description: Launch a Lakehouse Node and query sample data. --- diff --git a/advocacy_docs/edb-postgres-ai/analytics/reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx new file mode 100644 index 00000000000..85172f8b36d --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx @@ -0,0 +1,8 @@ +--- +title: Reference - EDB Postgres Lakehouse +navTitle: Reference +description: Things to know about EDB Postgres Lakehouse +--- + +Get started by querying sample data. + diff --git a/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx b/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx deleted file mode 100644 index 22dda70a721..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/roadmap.mdx +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Roadmap - EDB Postgres AI Analytics -navTitle: Roadmap -description: Where are we going with all this? ---- - -This is an early product, and we want you to get excited about the performance and possibilities. However, there's still a lot of work to do, especially as it relates to laoding data... - -Here are some of the things we're planning to support: - -* Continuous replication -* On-premises deployment -* Private buckets -* "Bring Your Own Bucket" (not just set in each node) -* Catalogs - diff --git a/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx b/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx deleted file mode 100644 index d330a4d53e2..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/sample_datasets.mdx +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Sample Datasets - EDB Postgres AI Analytics -navTitle: Sample Datasets -description: What datasets are available by default to Lakehouse Nodes? ---- - -When you launch a Lakehouse Node, it can query pre-loaded sample datasets in a public bucket in the same region. - -These are the datasets: - -- TPC-H SF 1 -- TPC-H SF 10 -- TPC-H SF 100 -- TPC-H SF 1000 -- TPC-DS SF 1 -- TPC-DS SF 10 -- TPC-DS SF 100 -- TPC-DS SF 1000 -- Clickbench - -These are the default bucket names: - -- blahblah - diff --git a/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx b/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx deleted file mode 100644 index 355b1a336b3..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/storage_locations.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: Storage Locations - EDB Postgres AI Analytics -navTitle: Storage Locations -description: Storage Locations are locations in object storage containing Lakehouse Tables. ---- - -Storage Locations are locations in object storage where you store Lakehouse Tables. - diff --git a/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx b/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx deleted file mode 100644 index 812497ff61e..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/sync_to_lakehouse.mdx +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Sync to Lakehouse - EDB Postgres AI Analytics -navTitle: Sync to Lakehouse -description: Learn how to sync data from a transactional database to Lakehouse Tables in object storage. ---- - -This guide will teach you how to sync data from a transactional database to Lakehouse Tables in a Managed Storage Location. - -## Assumptions and Goal - -We have some tables in a transactional database, and we want to sync them to Lakehouse Tables in a Managed Storage Location. - -Caveats, source database: - -* Must be a Postgres cluster in EDB Postgres AI Cloud Database -* Must be a managed Postgres AI Cloud Database -* Must be in AWS -* Must be in an EDB Hosted AWS region (not "Bring Your Own Account") - -Caveats, managed storage location: - -* Must be in the same project -* Must be in the same EDB Hosted AWS region -* Must be a _Managed_ Storage location - support for "bring your own bucket" destinations is coming soon... -* You cannot delete data in the Managed Storage Location - -Limitations: - -* One time sync -* Updating data requires performing another Sync to Lakehouse - -## Setup - -Create a transactional database... single node okay for this demo... - -## Create the migration - -1. Go to the cluster -2. Create the migration -3. Wait for it to complete - -## Start a Lakehouse Node - -1. Start Lakehouse Node -2. Query tables as defined in the migration - -## Start another Lakehouse Node - -Any Lakehouse Node in the project can query any Lakehouse Tables in the project. - diff --git a/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx b/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx deleted file mode 100644 index 7b7bf2b197b..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/terminology.mdx +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Terminology - EDB Postgres AI Analytics -navTitle: Terminology -description: The Lakehouse lexicon and terms to learn about analytics on EDB Postgres AI. ---- - -This terminology list includes terms associated with Analytics on EDB Postgres AI. - -### Lakehouse - -Lakehouse - -### Storage Location - -An organizational concept... - -#### Managed Storage Location - -A storage location where we manage the layout for you \ No newline at end of file diff --git a/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx b/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx deleted file mode 100644 index d198c3c15dc..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/user_management.mdx +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: User Management - EDB Postgres AI Analytics -navTitle: User Management -description: Manage access to Lakehouse Nodes ---- - -For access to Postgres AI Console, see - -Otherwise, access to Lakehouse Nodes is governed by the same rules as Postgres user management. - -## edb_admin - -When provisioning a cluster, a user `edb_admin` is created... - -This is the default - -### Creating users - -You can use Postgres user management to create users - -### Gotchas and Limitations - -When tables are re-created, grants will be lost... diff --git a/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx b/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx deleted file mode 100644 index 1c1fb722e41..00000000000 --- a/advocacy_docs/edb-postgres-ai/analytics/what_to_expect_when_querying.mdx +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: What to Expect When Querying - EDB Postgres AI Analytics -navTitle: What to Expect -description: In general, queries are "read only." All SELECT statements will work, but you cannot manipulate tables directly (at least not yet). ---- - -Here's what to expect.... \ No newline at end of file From df58bd700813cda4bc742d95abdec4f1bff01bdd Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Wed, 22 May 2024 22:39:27 +0100 Subject: [PATCH 62/67] Update advocacy cards for Analytics to use correct links + Lakehouse Settle on a middle ground of titling it "Lakehouse Analytics" The product name is Lakehouse, the category is Analytics --- src/pages/index.js | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/pages/index.js b/src/pages/index.js index b5b48f0d147..72e5309c5e6 100644 --- a/src/pages/index.js +++ b/src/pages/index.js @@ -195,18 +195,18 @@ const Page = () => { - - Getting Started + + Concepts - - Lakehouse + + Quick Start - - Analytics + + Reference Date: Thu, 23 May 2024 00:35:14 +0100 Subject: [PATCH 63/67] Add overview page for analytics --- .../edb-postgres-ai/analytics/index.mdx | 66 +++++++++++++++++-- 1 file changed, 62 insertions(+), 4 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 53ef24a8470..7ce64bd6f41 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -1,6 +1,6 @@ --- -title: EDB Postgres AI Analytics -navTitle: Analytics +title: Lakehouse Analytics +navTitle: Lakehouse Analytics indexCards: simple iconName: Improve navigation: @@ -9,7 +9,65 @@ navigation: - reference --- -Introduction to EDB Postgres AI Analytics. +_EDB Postgres Lakehouse extends the power of Postgres to analytical workloads, +by adding a vectorized query engine and separating storage from compute. Building +a Data Lakehouse has never been easier – just use Postgres._ -_EDB Postgres AI Analytics provides the tools needed to gather, analyze, and visualize data. It includes a variety of tools and features that help you to analyze data and make data-driven decisions. EDB Postgres AI Analytics is designed to help you to analyze data and make data-driven decisions._ +## Rapid Analytics for Postgres + +Postgres Lakehouse is a core offering of the EDB Postgres® AI platform, extending +Postgres to support analytical queries over columnar data in object storage, +while keeping the simplicity and ease of use that Postgres users love. + +With Postgres Lakehouse, you can query your Postgres data with a Lakehouse Node, +an ephemeral, scale-to-zero compute resource powered by Postgres that’s optimized for +vectorized query execution over columnar data. + +## Postgres Native + +Never leave the Postgres ecosystem. + +Postgres Lakehouse nodes run either EPAS or PGE as the Postgres engine, with +data for analytics stored as columnar tables in object storage using the +open source Delta Lake protocol. + +EDB Postgres Lakehouse is “just Postgres” – you can query it with any Postgres +client, and it fully supports all Postgres queries, functions and statements, so +there’s no need to change existing queries or reconfigure business +intelligence software. + +## Vectorized Execution + +Postgres Lakehouse uses Apache DataFusion’s vectorized SQL query engine to +execute analytical queries 5-100x faster (30x on average) compared to native +Postgres, while still falling back to native execution when necessary. + +## Columnar Storage + +Postgres Lakehouse is optimized to query "Lakehouse Tables" in object storage, +extending the power of open source database to open table formats. Currently, +it supports querying "Delta Tables" stored according to the Delta Lake protocol. + +## Lakehouse Sync + +You can sync your own data from tables in transactional sources (initially, EDB +Postgres AI Cloud Service databases) into Lakehouse Tables in Storage Locations +(initially, managed locations in S3 object storage). + +## Fully Managed Service + +You can launch Postgres Lakehouse nodes using the EDB Postgres AI Cloud +Service (formerly EDB BigAnimal). Point a Lakehouse Node at a storage bucket +with some Delta Tables in it, and get results of analytical (OLAP) queries in +less time than if you queried the same data in a transactional Postgres database. + +Postgres Lakehouse nodes are available now for customers using +EDB Postgres AI - Hosted environments on AWS, and will be rolling out +to additional cloud environments soon. + +## Try Today + +It’s easy to start using Postgres Lakehouse. Provision a Lakehouse Node in five +minutes, and start qureying pre-loaded benchmark data like TPC-H, TPC-DS, +Clickbench, and the 1 Billion Row challenge. From 511838706e767b67bfb9f44aeadf8407fd9aecba Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Thu, 23 May 2024 03:39:16 +0100 Subject: [PATCH 64/67] Add reference page --- .../edb-postgres-ai/analytics/index.mdx | 6 +- .../edb-postgres-ai/analytics/reference.mdx | 184 +++++++++++++++++- 2 files changed, 186 insertions(+), 4 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/analytics/index.mdx b/advocacy_docs/edb-postgres-ai/analytics/index.mdx index 7ce64bd6f41..d500f685131 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/index.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/index.mdx @@ -27,9 +27,9 @@ vectorized query execution over columnar data. Never leave the Postgres ecosystem. -Postgres Lakehouse nodes run either EPAS or PGE as the Postgres engine, with -data for analytics stored as columnar tables in object storage using the -open source Delta Lake protocol. +Postgres Lakehouse nodes run either EDB Advanced Server (EPAS) or EDB Postgres +Extended (PGE) as the Postgres engine, with data for analytics stored as +columnar tables in object storage using the open source Delta Lake protocol. EDB Postgres Lakehouse is “just Postgres” – you can query it with any Postgres client, and it fully supports all Postgres queries, functions and statements, so diff --git a/advocacy_docs/edb-postgres-ai/analytics/reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx index 85172f8b36d..02bf3ee5d3c 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/reference.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx @@ -4,5 +4,187 @@ navTitle: Reference description: Things to know about EDB Postgres Lakehouse --- -Get started by querying sample data. +Postgres Lakehouse is an early product. Eventually, it will support deployment +modes across multiple clouds and on-premises. However, currently it's fairly +limited in terms of where you can deploy it and what data you can query with it. +To get the best experience with Postgres Lakehouse, you should follow the +"quick start" guide to query benchmarking data. Then you can try loading your +own data with Lakehouse Sync. If you're intrigued, please reach out to us and +we can talk more about your use case and potential opportunities. + +This page details some of the important bits to know. + +## Supported Cloud Providers and Regions + +**AWS Only**: Currently, support for all Lakehouse features (Lakehouse Nodes, +Managed Storage Locations, and Lakehouse Sync) is limited to AWS. + +**EDB-Hosted Only**: "Bring Your Own Account" (BYOA) regions are NOT currently +supported for Lakehouse resources. Support is limited to +ONLY **EDB Postgres AI - Hosted** environments on AWS (a.k.a. "EDB-Hosted AWS regions"). + +This means you can select from one of the following regions: + +* North America + * US East 1 + * US East 2 + * US West 2 +* Europe + * EU Central 1 + * EU West 1 + * EU West 2 +* Asia + * AP South 1 +* Australia + * AP SouthEast 2 + +To be precise: + +* Lakehouse Nodes can only be provisioned in EDB-hosted AWS regions +* Managed Storage Locations can only be created in EDB-hosted AWS regions +* Lakehouse Sync can only sync from source databases in EDB-hosted AWS regions + +These limitations will be removed as we continue to improve the product. Eventually, +we will support BYOA, as well as Azure and GCP, for all Lakehouse use cases. We +will also add better support for "external" buckets ("bring your own bucket"). + +## Supported AWS Instances + +When deploying a Lakehouse Node, you must choose an instance type from +the `m6id` family of instances. Importantly, these instances come with NVMe +drives attached to them. + +**Instances are ephemeral.** These NVMe drives are used only for spill-out space +*while processing queries, and for caching Delta Tables on disk. +All data on the NVMe drives will be lost when the cluster is shutdown. + +**System tables are persisted.** Persistent data in system tables (users, roles, +*etc.) is stored in an attached +block storage device, and will survive a pause/resume cycle. + +**Supported Instances** + +| API Name | Memory | vCPUs | Cores | Storage | +| --------------- | --------- | --------- | ----- | ------------------------------- | +| `m6id.large` | 8.0 GiB | 2 vCPUs | 1 | 118 GB NVMe SSD | +| `m6id.xlarge` | 16.0 GiB | 4 vCPUs | 2 | 237 GB NVMe SSD | +| `m6id.2xlarge` | 32.0 GiB | 8 vCPUs | 4 | 474 GB NVMe SSD | +| `m6id.4xlarge` | 64.0 GiB | 16 vCPUs | 8 | 950 GB NVMe SSD | +| `m6id.8xlarge` | 128.0 GiB | 32 vCPUs | 16 | 1900 GB NVMe SSD | +| `m6id.12xlarge` | 192.0 GiB | 48 vCPUs | 24 | 2850 GB (2 \* 1425 GB NVMe SSD) | +| `m6id.16xlarge` | 256.0 GiB | 64 vCPUs | 32 | 3800 GB (2 \* 1900 GB NVMe SSD) | +| `m6id.24xlarge` | 384.0 GiB | 96 vCPUs | 48 | 5700 GB (4 \* 1425 GB NVMe SSD) | +| `m6id.32xlarge` | 512.0 GiB | 128 vCPUs | 64 | 7600 GB (4 \* 1900 GB NVMe SSD) | + +## Available Benchmarking Datasets + +When you provision a Lakehouse Node, it comes pre-configured to point to a public +S3 bucket in its same region, containing sample benchmarking datasets. + +You can query tables in these datasets by referencing them with their schema +name. + +| Schema Name | Dataset | +| --------------- | ---------------------------- | +| `tpcds_sf_1` | TPC-DS, Scale Factor 1 | +| `tpcds_sf_10` | TPC-DS, Scale Factor 10 | +| `tpcds_sf_100` | TPC-DS, Scale Factor 100 | +| `tpcds_sf_1000` | TPC-DS, Scale Factor 1000 | +| `tpch_sf_1` | TPC-H, Scale Factor 1 | +| `tpch_sf_10` | TPC-H, Scale Factor 10 | +| `tpch_sf_100` | TPC-H, Scale Factor 100 | +| `tpch_sf_1000` | TPC-H, Scale Factor 1000 | +| `clickbench` | ClickBench, 100 million rows | +| `brc_1b` | Billion row challenge | + +!!!note Notes about ClickBench data: + +Data columns (`EventData`) are integers, not dates. + +You must quote ClickBench column names, because they contain uppercase letters, +but unquoted identifiers in Postgres are case-insensitive. For example: + +✅ `select "Title" from clickbench.hits;` + +🚫 `select Title from clickbench.hits;` +!!! + +## Gotcha: Do not set `search_path` + +Do not set `search_path`. Always reference fully qualified table names. + +Using `search_path` makes Postgres Lakehouse fall back to PostgreSQL, +dramatically impacting query performance. To avoid this, qualify all table names +in your query with a schema. + +For example: + +**🚫 Do NOT do this!** + +```sql +--- DO NOT DO THIS +SET search_path = tpch_sf_10; +SELECT COUNT(*) FROM lineitem; +``` + +**✅ Do this instead!** + +```sql +SELECT COUNT(*) FROM tpch_sf_10.lineitem +``` + +## DirectScan vs. Fallback Modes and EXPLAIN + +Postgres Lakehouse is fastest when it can "push down" your entire query to +DataFusion, the vectorized query used for handling queries when possible. (In the +future, this will be more fine-grained as we add support for partial pushdowns.) + +Postgres Lakehouse can execute your query in two modes. First, it attempts to +run the entire query using Seafowl (a dedicated columnar database based on +DataFusion). If Seafowl can’t run the entire query, for example, because it +uses PostgreSQL-specific operations like JSON, then Postgres Lakehouse will fall +back to using the PostgreSQL executor, with Seafowl streaming full table +contents to it. + +If your query is extremely slow, it’s possible that’s what’s happening. + +You can check which mode is being used by running an `EXPLAIN` on the query and +making sure that the top-most query node is `SeafowlDirectScan`. For example: + +``` +explain select count from (select count(*) from tpch_sf_1.lineitem); + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ + Aggregate (cost=167.52..167.55 rows=1 width=8) + -> Append (cost=0.00..165.01 rows=1001 width=0) + -> Seq Scan on lineitem lineitem_1 (cost=0.00..0.00 rows=1 width=0) + -> SeafowlScan on "16529" lineitem_2 (cost=100.00..150.00 rows=1000 width=0) + SeafowlPlan: logical_plan + TableScan: tpch_sf_1.lineitem projection=[l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate, l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment] +(6 rows) +``` + + +In this case, the query is executed by PostgreSQL and Seafowl is only involved +when scanning the table (see `SeafowlScan` at the bottom). The fix in this case is +to explicitly name the inner `COUNT(*)` column, since Seafowl gives it an implicit +name `count(*)` whereas PostgreSQL calls it `count`: + + +``` +edb_admin=> explain select count from (select count(*) as count from tpch_sf_1.lineitem); + QUERY PLAN +-------------------------------------------------------------------- + SeafowlDirectScan: logical_plan + Projection: COUNT(*) AS count + Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1)) AS COUNT(*)]] + TableScan: tpch_sf_1.lineitem projection=[] +(4 rows) +``` + +Here, we can see the `SeafowlDirectScan` at the top, which means that Seafowl is +running the entire query. + +If you’re having trouble rewording your query to make it run fully on Seafowl, +please open a support ticket. From 03a2e781b10de88fbd434e4a5594b6d7d80002fe Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Thu, 23 May 2024 04:04:27 +0100 Subject: [PATCH 65/67] Add more sections to reference (loading data) --- .../edb-postgres-ai/analytics/reference.mdx | 111 ++++++++++++++++++ 1 file changed, 111 insertions(+) diff --git a/advocacy_docs/edb-postgres-ai/analytics/reference.mdx b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx index 02bf3ee5d3c..47247f5be73 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/reference.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/reference.mdx @@ -110,6 +110,14 @@ but unquoted identifiers in Postgres are case-insensitive. For example: 🚫 `select Title from clickbench.hits;` !!! +## User Management + +When you provision a Lakehouse Node, you must provide a password. We do not +save this password. You will need it to login as the `edb_admin` user. This is +not a superuser account, but it does have the ability to create users and roles +and grants. Thus, you can either share the credentials for `edb_admin` itself, +or you can create other users and distribute those. + ## Gotcha: Do not set `search_path` Do not set `search_path`. Always reference fully qualified table names. @@ -134,6 +142,24 @@ SELECT COUNT(*) FROM lineitem; SELECT COUNT(*) FROM tpch_sf_10.lineitem ``` +## Supported Queries + +In general, **READ ONLY** queries are supported. You cannot write directly to +object storage. This includes all Postgres built-in functions, statements +and types. It also includes any of those provided by EPAS or PGE, depending on +which distribution you choose to deploy. + +In general, you cannot insert, update, delete or otherwise modify data. You +cannot `CREATE TABLE`. You must load data into the bucket out-of-band, either +with your own ETL scripts or with Lakehouse Sync. See "Advanced: Bring Your Own +Data" for more details. (In the future, we will be making this more usable with +a custom DDL). + +One exception is Postgres system tables, such as those used for storing users, +roles, and grants. These tables are stored on the local block device, which is +included in backups and restores. So you can `CREATE USER` or `CREATE ROLE` or +`GRANT USAGE`, and these users/roles/grants will survive restarts and restores. + ## DirectScan vs. Fallback Modes and EXPLAIN Postgres Lakehouse is fastest when it can "push down" your entire query to @@ -188,3 +214,88 @@ running the entire query. If you’re having trouble rewording your query to make it run fully on Seafowl, please open a support ticket. + +## Load Data with Lakehouse Sync + +If you have a transactional database running in EDB Postgres AI Cloud Service, +then you can sync tables from this database into a Managed Storage Location. + +A more detailed guide for this is forthcoming. If you want to try it yourself, +look in the UI for "Migrations" or "Sync to Lakehouse." + +## Advanced: Bring Your Own Data + +It's possible to point your Lakehouse Node at an arbitrary S3 bucket with Delta +Tables inside of it. However, this comes with some major caveats (which will +eventually be resolved): + +### Caveats + +* The bucket must be publicly accessible. + * If you want to use a private bucket, this is technically possible, but +requires some manual action on our side and your side (to assign the correct +IAM policies). Please let us know if you want to try it. We will be adding +proper support for private, external buckets in the near future. +* The tables must be stored as [Delta Tables](http://github.com/delta-io/delta/blob/master/PROTOCOL.md) within the location +* A “Delta Table” is a folder of Parquet files along with some JSON metadata. +* Each table must be prefixed with a `$schema/$table/` where `$schema` and `$table` are valid Postgres identifiers (i.e. < 64 characters) + * For example, this is a valid Delta Table that will be recognized by Beacon Analytics: + * `my_schema/my_table/{part1.parquet, part2.parquet, _delta_log}` + * These `$schema` and `$table` identifiers will be queryable in the Lakehouse node, e.g.: + * `SELECT count(*) FROM my_schema.my_table;` + * This Delta Table will NOT be recognized by Beacon Analytics (missing a schema): + * `my_table/{part1.parquet, part2.parquet, _delta_log}` + + +### Loading your own data + +* You can use the [deltalake](https://pypi.org/project/deltalake/) Python library +to create Delta Tables and write to the bucket +* You can also use the [`lakehouse-loader`](https://github.com/splitgraph/lakehouse-loader) utility +we created for this, to export data from an arbitrary Postgres instance to Lakehouse Tables +in a storage bucket. + +For example, with the `lakehouse-loader` utility: + +```bash +export PGPASSWORD="..." +export AWS_ACCESS_KEY_ID="..." +export AWS_SECRET_ACCESS_KEY="..." +# export other AWS envvars + +./lakehouse-loader postgres-to-delta postgres://test-user@localhost:5432/test-db -q "SELECT * FROM some_table" s3://my-bucket/my_schema/my_table +``` + +### Pointing to your bucket + +By default, each Lakehouse Node is configured to point to a bucket with +benchmarking datasets inside. To point it to a different bucket, you can +call the `seafowl.set_bucket_location` function: + +```sql +SELECT seafowl.set_bucket_location('{"region": "ap-south-1", "bucket": "my-bucket", "public": true}'); +``` + +### Querying your own data + +In the example above, after you've called `set_bucket_location`, you will be able +to query data in `my_schema.my_table`: + +```sql +SELECT * FROM some_table; +``` + +Note that using an S3 bucket that isn’t in the same region as your node +will 1) be slow because of cross-region latencies, and 2) will incur +AWS costs (between $0.01 and $0.02 / GB) for data transfer! Currently these +egress costs are not passed through to you but we do track them and reserve +the right to terminate an instance. + +### Switching back to sample data + +To switch the bucket back to the default sample bucket in the same region as your node: + +```sql +SELECT seafowl.set_bucket_location(NULL) +``` + From b7531df3da22b8ec0882960fd73b442ac73e3b8c Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Thu, 23 May 2024 04:27:21 +0100 Subject: [PATCH 66/67] Add quick start guide for Postgres Lakehouse --- .../images/create-cluster-dropdown.png | 3 + .../images/create-cluster-wizard.png | 3 + .../images/level-300-architecture.png | 3 + .../edb-postgres-ai/analytics/quick_start.mdx | 162 +++++++++++++++++- 4 files changed, 170 insertions(+), 1 deletion(-) create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-dropdown.png create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-wizard.png create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/level-300-architecture.png diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-dropdown.png b/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-dropdown.png new file mode 100644 index 00000000000..0b798360ddc --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-dropdown.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:861f6e7c7b2c03c21fc6fd9d0c6e5f48b8304cfbc02d522fe86b011b63ddd1a7 +size 99124 diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-wizard.png b/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-wizard.png new file mode 100644 index 00000000000..cdc0cd271bf --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/create-cluster-wizard.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:294c1a1fa3563bd2634d266b38f9f3ee647cbb6622ce1cb46632d962cd0f7281 +size 425337 diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/level-300-architecture.png b/advocacy_docs/edb-postgres-ai/analytics/images/level-300-architecture.png new file mode 100644 index 00000000000..ea69945d1ff --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/level-300-architecture.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:92662f8a9ba963b7d5a590583f4751e65738e8ac69ca70067b1d47e0b09f6dd2 +size 458380 diff --git a/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx index 8831edabd21..ecebbb65c03 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/quick_start.mdx @@ -4,5 +4,165 @@ navTitle: Quick Start description: Launch a Lakehouse Node and query sample data. --- -Get started by querying sample data. +In this guide, you will: +1. Create a Lakehouse Node +2. Connect to the node with your preferred Postgres client +3. Query sample data (TPC-H, TPC-DS, Clickbench, or 1BRC) in object storage + +For more details and advanced use cases, see [reference](./reference). + +## Introduction + +Postgres Lakehouse is a new type of Postgres “cluster” (it’s really just one +node) that you can provision in EDB Postgres AI Cloud Services (formerly known +as "BigAnimal"). It includes a vectorized query engine (based on Apache +[DataFusion](https://github.com/apache/datafusion)) for fast queries over +columnar “Lakehouse Tables” in object storage +(using [Delta Lake protocol](https://delta.io/)). + +### What to know + +For more details, see [concepts](./concepts) and [reference](./reference), +but this should cover the gist of it: + +* Postgres Lakehouse nodes are ephemeral, but they have NVMe hard drives for +local cache and spill out space. Currently we support deploying them to +EDB-hosted regions in AWS. +* Postgres Lakehouse nodes are pre-configured to query benchmark datasets +in object storage. +* Postgres Lakehouse is generally “read-only,” but otherwise supports all +valid Postgres queries. +* System tables are writable, so you can create users and roles. These tables +are stored on the attached block storage, so they will persist between instance +restarts and will be saved as part of backup/restore operations. Otherwise, +Lakehouse tables will not be part of backups, since they are ultimately stored +in object storage. + +### Basic Architecture + +Here's "what's in the box of a Lakehouse Node: + +![Level 300 Architecture of Postgres Lakehouse Node](./images/level-300-architecture.png) + +## Getting Started + +You will need an EDB Postgres AI account. Once you’ve logged in and created +a project, you can create a cluster. + +### Create a Lakehouse Node + +You will see a “Lakehouse Analytics” option under the “Create New” dropdown +on your project page: + +![Create Lakehouse Node Dropdown](./images/create-cluster-dropdown.png) + +Clicking this button will start a configuration wizard that looks like this: + +![Create Lakehouse Node Wizard Step 1](./images/create-cluster-wizard.png) + +Provisioning the cluster may take 10-15 minutes. + +A few notes: + +* You will need to provide your own password. Create a secure password and save +it to your password manager. We will not save it on our side. +* Only AWS is supported, in BigAnimal Hosted regions. +* You can choose EDB Postgres Advanced Server (EPAS) or EDB Postgres Extended (PGE), +and the pricing calculator uses the same formula with license costs as it +would for a normal EPAS or PGE single-node cluster. +* Only instances with NVMe drives are supported. This is the M6ID family. +The NVMe drives are used for spill-out space while processing queries. +All data on the NVMe drives will be lost on cluster shutdown. +Persistent data in system tables (users, roles, etc) is stored in an attached +block storage device and will survive a restart or backup/restore cycle. +* Only Postgres 16 is supported. + +For more notes about supported instance sizes, +see [reference - supported AWS instances](./reference/#supported-aws-instances). + +## Operating a Lakehouse Node + +### Connect to the Node + +You can connect to the Lakehouse Node with any Postgres client, in the same way +that you connect to any other cluster from EDB Postgres AI Cloud Service +(formerly known as BigAnimal): navigate to the cluster detail page and copy its +connection string. + +For example, you might copy the `.pgpass` blob into `~/.pgpass` (making sure to +replace `$YOUR_PASSWORD` with the password you provided when launching the +cluster). Then you can copy the connection string and use it as an argument to +`psql` or `pgcli`. + +In general, you should be able to connect to the database with any Postgres +client. We expect all introspection queries to work, and if you find one that +does not, then that is a bug. + +### Understand the constraints + +* Every cluster uses EPAS or PGE. So expect to see boilerplate tables from those +flavors in the installation when you connect. +* Queryable data (like the benchmarking datasets) is stored in object storage +as Delta Tables. Every cluster comes pre-loaded to point to a storage bucket +with benchmarking data inside (TPC-H, TPC-DS, Clickbench) at +scale factors 1 and 10. +* Only AWS is supported at the moment. Bring Your OWn Account (BYOA) is not supported. +* You can deploy a cluster in any region that is activated in +your EDB Postgres AI Account. Each region has a bucket with a copy of the +benchmarking data, and so when you launch a cluster, it will use the +benchmarking data in the location closest to it. +* The cluster is ephemeral. None of the data is stored on the hard drive, +except for data in system tables, e.g. roles and users and grants. +If you restart the cluster, or backup the cluster and then restore it, +it will restore these system tables. But the data in object storage will +remain untouched. +* The cluster supports READ ONLY queries of the data in object +storage (but it supports write queries to system tables for creating users, +etc.). You cannot write directly to object storage. You cannot create new tables. +* If you want to load your own data into object storage, +see [reference - bring your own data](./reference/#advanced-bring-your-own-data). + +## Inspect the Benchmark Datasets + +Inspect the Benchmark Datasets. Every cluster has some benchmarking data +available out of the box. If you are using pgcli, you can run `\dn` to see +the available tables. + +The available benchmarking datsets are: + +* TPC-H, at scale factors 1, 10, 100 and 1000 +* TPC-DS, at scale factors 1, 10, 100 and 1000 +* Clickbench, 100 million rows +* 1 Billion Row Challenge + +For more details on benchmark datasets, +see [reference - available benchmarking datasets](./reference/#available-benchmarking-datasets). + +## Query the Benchmark Datasets + +You can try running some basic queries: + +```sql +edb_admin> select count(*) from clickbench.hits; ++----------+ +| count | +|----------| +| 99997497 | ++----------+ +SELECT 1 +Time: 0.945s + + +edb_admin> select count(*) from brc_1b.measurements; ++------------+ +| count | +|------------| +| 1000000000 | ++------------+ +SELECT 1 +Time: 0.651s +``` + +Note: Do not use `search_path`! Please read the [reference](./reference) +page for more gotchas and information about syntax/query compatibility. From dce614ecb3a291a8e3c594b59f8ae5ab61b7ed67 Mon Sep 17 00:00:00 2001 From: Miles Richardson Date: Thu, 23 May 2024 04:59:20 +0100 Subject: [PATCH 67/67] Add analytics concepts page --- .../edb-postgres-ai/analytics/concepts.mdx | 119 +++++++++++++++++- .../images/level-100-architecture.png | 3 + .../images/level-200-marketecture.png | 3 + .../images/level-50-architecture.png | 3 + 4 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/level-100-architecture.png create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/level-200-marketecture.png create mode 100644 advocacy_docs/edb-postgres-ai/analytics/images/level-50-architecture.png diff --git a/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx b/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx index 3380f3e31d4..1dbd7a1becc 100644 --- a/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx +++ b/advocacy_docs/edb-postgres-ai/analytics/concepts.mdx @@ -4,4 +4,121 @@ navTitle: Concepts description: Learn about the ideas and terminology behind EDB Postgres Lakehouse for Analytics workloads. --- -Concepts about stuff. \ No newline at end of file +EDB Postgres Lakehouse is the solution for running Rapid Analytics against +operational data on the EDB Postgres® AI platform. + +## Major Concepts + +* **Lakehouse Nodes** query **Lakehouse Tables** in **Managed Storage Locations**. +* **Lakehouse Sync** can create **Lakehouse Tables** from **Transactional Tables** in a source database. + +Here's how it fits together: + +![Level 50 basic architecture](./images/level-50-architecture.png) + +### Lakehouse Node + +A Postgres Lakehouse Node is Postgres, with a Vectorized Query Engine that's +optimized to query Lakehouse Tables, but still fall back to Postgres for full +compatibility. + +Lakehouse nodes are stateless and ephemeral. Scale them up or down based on +workload requirements. + +### Lakehouse Tables + +Lakehouse Tables are stored using highly compresible, columnar storage formats +optimized for analytics and interoperable with the rest of the Analytics ecosystem. +Currently, Postgres Lakehouse Nodes can read tables stored using the Delta +Protocol ("delta tables"), and Lakehouse Sync can write them. + +### Managed Storage Location + +A Managed Storage Location is where you can organize Lakehouse Tables in +object storage, so that Postgres Lakehouse can query them. + +A "Managed Storage Location" is a location in object storage where we control +the file layout and write Lakehouse Tables on your behalf. Technically, it's an +implementation detail that we store these in buckets. This is really a subset +of an upcoming "Storage Location" feature that will also support +"External Storage Locations," where you bring your own bucket. + +### Lakehouse Sync + +Lakehouse Sync is a Data Migration Service offered as part of the EDB +Postgres® AI platform. It can "sync" tables from a transactional database, to +Lakehouse Tables in a destination Storage Location. Currently, it supports +source databases hosted in the EDB Postgres AI Cloud Service (formerly known as +BigAnimal). In the future, it will support more source databases (such as Oracle) +and deployment locations (on-premises databases). +It's built using [Debezium](https://debezium.io). + +## Terminology + +### Lakehouse + +The +"[Lakehouse Architecture](https://15721.courses.cs.cmu.edu/spring2023/papers/02-modern/armbrust-cidr21.pdf)" +is a data engineering practice, which is a portmanteau of "Data _Lake_" and "Data +Ware_house_," offering the best of both. The central tenet of the architecture is +that data is stored in Object Storage, generally in columnar formats like +Parquet, where different query engines can process it for their own specialized +purposes, using the optimal compute resources for a given query. + +### Vectorized Query Engine + +A vectorized query engine is a query engine that's optimized for running queries +on columnar data. Most analytics engines use vectorized query execution. +Postgres Lakehouse uses [Apache DataFusion](https://datafusion.apache.org/). + +### Delta Tables + +We use the term "Lakehouse Tables" to avoid overcommitting to a particular +format (since we might eventually support Iceberg or Hudi, for example). But +technically, we're using [Delta Tables](https://delta.io/). A Delta Table +is a well-defined container of Parquet files and JSON metadata, according to +the "Delta Lake" spec and open protocol. Delta Lake is a Linux Foundation project. + +## How it Works + +Postgres Lakehouse is built using a number of technologies: + +* PostgreSQL +* [Seafowl](https://seafowl.io/), an analytical database acquired from Splitgraph +* [Apache DataFusion](https://datafusion.apache.org/), the query engine used by Seafowl +* [Delta Lake](https://delta.io) (and specifically [delta-rs](https://github.com/delta-io/delta-rs)), +for implementing the storage and retrieval layer of Delta Tables + +Postgres Lakehouse is built by much of the Splitgraph team, along with Postgres +experts from EDB. We have lots of exciting plans for further integration with +existing EDB technologies... + +### Level 100 + +The most important thing to understand about Postgres Lakehouse, is that it +separates storage from compute. This allows you to scale them independently, +which is ideal for analytical workloads where queries can be unpredictable and +spiky. You would not want to keep a machine mostly idle just to hold data on +its attached hard drives. Instead, you can keep data in object storage (and also +in highly compressible formats), and only provision the compute needed to query +it when necessary. + +![Level 100 Architecture](./images/level-100-architecture.png) + +On the compute side, a Vectorized Query Engine is optimized to query Lakehouse +Tables, but still fall back to Postgres for full compatibility. + +On the storage side, Lakehouse Tables are stored using highly compressible, +columnar storage formats optimzied for Analytics. + +### Level 200 + +Here's a slightly more comprehensive diagram of how these services fit together: + +![Level 200 Marketecture](./images/level-200-marketecture.png) + +### Level 300 + +Here's the more detailed, zoomed-in view of "what's in the box": + +![Level 300 Architecture](./images/level-300-architecture.png) diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/level-100-architecture.png b/advocacy_docs/edb-postgres-ai/analytics/images/level-100-architecture.png new file mode 100644 index 00000000000..f38ceb9aa30 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/level-100-architecture.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4aef8790c4aa13f37492cf6acb1c1376a7db86bde75d5fa403446280594b70c2 +size 309502 diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/level-200-marketecture.png b/advocacy_docs/edb-postgres-ai/analytics/images/level-200-marketecture.png new file mode 100644 index 00000000000..65d48afbe42 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/level-200-marketecture.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:39a0b65fac78b1314645cb31cc0b1ab7357e55f9338e97b76e5cf6a20a9f94dd +size 453185 diff --git a/advocacy_docs/edb-postgres-ai/analytics/images/level-50-architecture.png b/advocacy_docs/edb-postgres-ai/analytics/images/level-50-architecture.png new file mode 100644 index 00000000000..fb983b28acd --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/analytics/images/level-50-architecture.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:079cecc1a546705681e2a0a302680172726195aead9d6f63e49ecf702326a0e9 +size 436789