OOM
#2273
Replies: 1 comment
-
jdbcProperties.setProperty("scan.incremental.snapshot.chunk.size", "1024"); DatabaseSetting fromDatabaseSetting = drsSetting.getFromDatabaseSetting();
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
一次性读取数据太多,造成OOM。按照官网说明在flink-conf.yaml设置了下面的参数也没用
The chunk size (number of rows) of table snapshot, captured tables are split into multiple chunks when read the snapshot of table.
scan.incremental.snapshot.chunk.size: 2048
he maximum fetch size for per poll when read table snapshot.
scan.snapshot.fetch.size: 256
cdc一次读取的数据行数
cdc.reader.fetch-size: 2000
错误信息如下,还需要什么设置不。
2023-07-07 14:47:00,579 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 225685 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:19.86
2023-07-07 14:47:05,652 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 114498 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:24.933
2023-07-07 14:47:11,803 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 285931 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:31.082
2023-07-07 14:47:17,222 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 154307 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:36.503
2023-07-07 14:47:22,892 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 306658 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:42.173
2023-07-07 14:47:28,384 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 163945 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:47.665
2023-07-07 14:47:33,909 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 311950 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:53.19
2023-07-07 14:47:39,390 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 166262 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:58.671
2023-07-07 14:47:45,214 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 313365 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:04.495
2023-07-07 14:47:52,457 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 167219 records for split 'egenie_baseinfo.platform_product_match:1' after 00:01:11.738
2023-07-07 14:47:57,005 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314004 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:16.286
2023-07-07 14:48:07,657 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314190 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:26.937
2023-07-07 14:48:07,657 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 167465 records for split 'egenie_baseinfo.platform_product_match:1' after 00:01:22.806
2023-07-07 14:48:18,978 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314290 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:38.258
2023-07-07 14:48:17,708 ERROR com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader [] - Execute snapshot read task for mysql split MySqlSnapshotSplit{tableId=egenie_baseinfo.platform_product_match, splitId='egenie_baseinfo.platform_product_match:1', splitKeyType=[
platform_product_match_id
BIGINT NOT NULL], splitStart=[12077194], splitEnd=[12482912], highWatermark=null} failjava.lang.OutOfMemoryError: GC overhead limit exceeded
Beta Was this translation helpful? Give feedback.
All reactions