From 477a3f4109e06e1c1ba9096d2c58bde83cf25581 Mon Sep 17 00:00:00 2001 From: Chris Cranford Date: Mon, 22 Jan 2024 11:46:13 -0500 Subject: [PATCH] Debezium 2.6.0.Alpha1 Announcement --- _data/releases/2.6/2.6.0.Alpha1.yml | 2 +- _includes/documentation.html | 6 +- ...24-01-22-debezium-2.6-alpha1-released.adoc | 121 ++++++++++++++++++ 3 files changed, 127 insertions(+), 2 deletions(-) create mode 100644 _posts/2024-01-22-debezium-2.6-alpha1-released.adoc diff --git a/_data/releases/2.6/2.6.0.Alpha1.yml b/_data/releases/2.6/2.6.0.Alpha1.yml index 50fe2f25176..cc9e95f37a6 100644 --- a/_data/releases/2.6/2.6.0.Alpha1.yml +++ b/_data/releases/2.6/2.6.0.Alpha1.yml @@ -2,5 +2,5 @@ date: 2024-01-18 version: "2.6.0.Alpha1" stable: false summary: Removal of `replica_set` mode for MongoDB connector; Notification improvements; CloudEvents schema naming customization; Support for `NEW_ROW_AND_OLD_VALUES` capture mode in Spanner connector; Support for service loader classloading mode in Kafka Connect; DDL grammar fixes; Upgrade to Groovy 4; Fixed retrying logic for Debezium Engine -#announcement_url: +announcement_url: /blog/2024/01/22/debezium-2-6-alpha1-released/ diff --git a/_includes/documentation.html b/_includes/documentation.html index 42a8a757276..c21dc5f1113 100644 --- a/_includes/documentation.html +++ b/_includes/documentation.html @@ -26,9 +26,13 @@

{{ version }}

{{ latestRelease.date }}
-
{{ latestRelease.summary }}
+
{{ latestRelease.summary | truncate: 120 }}
+ {% if labelText = 'development' %} + + {% else %} + {% endif %} More info     diff --git a/_posts/2024-01-22-debezium-2.6-alpha1-released.adoc b/_posts/2024-01-22-debezium-2.6-alpha1-released.adoc new file mode 100644 index 00000000000..40c68659686 --- /dev/null +++ b/_posts/2024-01-22-debezium-2.6-alpha1-released.adoc @@ -0,0 +1,121 @@ +--- +layout: post +title: Debezium 2.6.0.Alpha1 Released +date: 2024-01-22 +tags: [ releases, mongodb, mysql, postgres, sqlserver, cassandra, oracle, db2, vitess, outbox, spanner, jdbc ] +author: ccranfor +--- + +A new year, a new preview release, in true Debezium fashion. +The team is pleased to announce the first installment of the Debezium 2.6 release stream, **Debezium 2.6.0.Alpha1**. +Let's take a moment and dive into these new features, understand how to use these to improve your change data capture experience... + +++++++ + +== Breaking changes + +The team aims to avoid any potential breaking changes between minor releases; however, such changes are sometimes inevitable. + +MongoDB:: + +* The MongoDB connector no longer supports the `replica_set` mode (https://issues.redhat.com/browse/DBZ-7260)[DBZ-7260]). +This has been a feature that has been deprecated for several versions and there has been ongoing work over Debezium 2.x to achieve this goal. +If you are using the `replica_set` mode, you will need to make adjustments when using Debezium 2.6+. + +Re-select Columns Post Processor:: + +* The re-select columns post processor used the key based on the `message.key.columns` when building the query. This is not correct for most tables with primary keys. +The default behavior has changed and the table primary key is used by default. +A new configuration option was introduced to allow the user to choose between whether to use the primary key or the generated key, `reselect.use.event.key` (https://issues.redhat.com/browse/DBZ-7358[DBZ-7358]). + +== Improvements and changes + +=== New Matching Collections API added + +One of the team's ongoing tasks include the migration of Debezium UI's backend into the main Debezium repository. +One of the unique benefits with doing this is we can identify where there is code overlap between a connector's runtime and the UI, and develop interface contracts to expose this shared data. + +Thanks to a community contribution for https://issues.redhat.com/browse/DBZ-7167[DBZ-7167], the `RelationalBaseSourceConnector` contract has been adjusted and a new method introduced to return a list of table names that match the connector's specific configuration. +Any connector that implements this abstract base class will need to implement this new method. + +=== CloudEvents schema name customization + +When using schema registry, event schemas need to be registered with a name so that they can be looked up upon later inquiries by pipelines. +So when pairing CloudEvents formatted messages with schema registry, the same applies and in Debezium 2.6, you can explicitly control how the name is registered. + +By default, the schema for a CloudEvent message will be automatically generated by the converter. +However, if the auto generated schema names are not sufficient, you can adjust the configuration by specifying `dataSchemaName`, which can be set either to `generate` (the default behavior) or `header` to pull the schema name directly from the specified event header field. + +=== Oracle Infinispan cache improvements + +The Debezium Oracle connector maintains a buffer of all in-flight transactions, and this buffer can be allocated off-heap using Infinispan. +Sometimes, the user configuration specifies that if an in-flight transaction lasts longer than the specified number of milliseconds, the transaction can be abandoned or discarded by the buffer. +This means that the transaction will be forgotten and not emitted by the connector. + +In order to improve metrics integration with frameworks like Grafana and Prometheus, a new JMX metric, `AbandonedTransactionCount`, was added to track the number of transactions that are abandoned by the connector during it's runtime. + +=== Supports Spanner `NEW_ROW_AND_OLD_VALUES` value capture type + +Google Spanner's value capture type is responsible for controlling how the change stream represents the change data in the event stream and are configured when constructing the change stream. + +Spanner introduced a new value capture mode called `NEW_ROW_AND_OLD_VALUES`, which is responsible for capturing all values of tracked columns, both modified and unmodified, whenever any column changes. +This new mode is an improvement over `NEW_ROW` because it also includes the capture of old values, making it align with what you typically observe with other Debezium connectors. + +== Other changes + +Altogether, https://issues.redhat.com/issues/?jql=project%20%3D%20DBZ%20AND%20fixVersion%20%3D%202.6.0.Alpha1%20ORDER%20BY%20component%20ASC[25 issues] were fixed in this release: + +* Empty object sent to GCP Pub/Sub after DELETE event https://issues.redhat.com/browse/DBZ-7098[DBZ-7098] +* Notifications are Missing the ID field in log channel https://issues.redhat.com/browse/DBZ-7249[DBZ-7249] +* Debezium-ddl-parser crashes on parsing MySQL DDL statement (sub-query with UNION) https://issues.redhat.com/browse/DBZ-7259[DBZ-7259] +* Oracle DDL parsing error in PARTITION REFERENCE https://issues.redhat.com/browse/DBZ-7266[DBZ-7266] +* Enhance Oracle's CREATE TABLE for Multiple Table Specifications https://issues.redhat.com/browse/DBZ-7286[DBZ-7286] +* Add service loader manifests for all Connect plugins https://issues.redhat.com/browse/DBZ-7298[DBZ-7298] +* PostgreSQL ad-hoc blocking snapshots fail when snapshot mode is "never" https://issues.redhat.com/browse/DBZ-7311[DBZ-7311] +* Ad-hoc blocking snapshot dies with "invalid snapshot identifier" immediately after connector creation https://issues.redhat.com/browse/DBZ-7312[DBZ-7312] +* Specifying a table include list with spaces between elements cause LogMiner queries to miss matches https://issues.redhat.com/browse/DBZ-7315[DBZ-7315] +* Debezium heartbeat.action.query does not start before writing to WAL: part 2 https://issues.redhat.com/browse/DBZ-7316[DBZ-7316] +* Update Groovy version to 4.x https://issues.redhat.com/browse/DBZ-7340[DBZ-7340] +* errors.max.retries is not used to stop retrying https://issues.redhat.com/browse/DBZ-7342[DBZ-7342] +* Upgrade Antora to 3.1.7 https://issues.redhat.com/browse/DBZ-7344[DBZ-7344] +* Oracle connector is occasionally unable to find SCN https://issues.redhat.com/browse/DBZ-7345[DBZ-7345] +* Initial snapshot notifications should use full identifier. https://issues.redhat.com/browse/DBZ-7347[DBZ-7347] +* Upgrade Outbox Extension to Quarkus 3.6.5 https://issues.redhat.com/browse/DBZ-7352[DBZ-7352] +* MySqlJdbcSinkDataTypeConverterIT#testBooleanDataTypeMapping fails https://issues.redhat.com/browse/DBZ-7355[DBZ-7355] + +A big thank you to all the contributors from the community who worked on this release: +https://github.com/ani-sha[Anisha Mohanty], +https://github.com/shybovycha[Artem Shubovych], +https://github.com/roldanbob[Bob Roldan], +https://github.com/Naros[Chris Cranford], +https://github.com/ilyasahsan123[Ilyas Ahsan], +https://github.com/indraraj[Indra Shukla], +https://github.com/jcechace[Jakub Cechacek], +https://github.com/james-johnston-thumbtack[James Johnston], +https://github.com/sherpa003[Jiri Kulhanek], +https://github.com/jpechane[Jiri Pechanec], +https://github.com/mfvitale[Mario Fiore Vitale], +https://github.com/mimaison[Mickael Maison], +https://github.com/obabec[Ondrej Babec], +https://github.com/wukachn[Peter Hamer], +https://github.com/richardharrington[Richard Harrington], +https://github.com/roldanbob[Robert Roldan], +https://github.com/rkudryashov[Roman Kudryashov], +https://github.com/ShuranZhang[Shuran Zhang], +https://github.com/vsantona[Vincenzo Santonastaso], +https://github.com/vjuranek[Vojtech Juranek], and +https://github.com/overwatcheddude[حمود سمبول]! + +== Outlook & What's next? + +The Debezium 2.6 release cycle is one of our most ambitious initiatives with lots of new features and changes. +You can find more about what the team is working on specifically for 2.6 and the road to Debezium 3.0 in our link:/docs/roadmap[road map]. +If you have any suggestions or ideas, please feel free to get in touch with us on our https://groups.google.com/forum/#!forum/debezium[mailing list] or in our https://debezium.zulipchat.com/login/#narrow/stream/302529-users[Zulip chat]. + +As the team continues springing into action with Debezium 2.6, we also intend to continue to bug fix and address any regressions that are reported to last quarter's Debezium 2.5 release. +Debezium 2.5 is now the project's stable release, and we encourage everyone to upgrade and get the latest and greatest features. +In fact, you can expect the next maintenance release of Debezium, **2.5.1.Final** to be released later this week :). + +Until next time, happy streaming! + +