Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polling efficiency #378

Merged
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
29b360d
intial changes
Dec 27, 2024
22fa3a0
Merge remote-tracking branch 'origin/s3-source-release' into polling
Dec 27, 2024
b6f2d08
fixed test cases
Dec 27, 2024
38fe822
Merge branch 's3-source-release' into polling_efficiency
Dec 27, 2024
0a10606
Added more complete tests
Dec 30, 2024
d7b8236
Merge branch 's3-source-release' into polling_efficiency
Claudenw Dec 31, 2024
9ccb1a8
Migrate to new AWS client
Dec 31, 2024
d3ce578
added AWS Integration test
Dec 31, 2024
e744e32
attempts to fix polling
Jan 2, 2025
df11418
sped up integration test
Jan 2, 2025
be775b0
fixed tests
Jan 6, 2025
ebf0e8a
removed unused class
Jan 6, 2025
9c73cd4
removed unused methods
Jan 6, 2025
d7b2f3c
Changes to allow Backoff to abort the timer
Jan 6, 2025
cf32e79
updated javadoc
Jan 6, 2025
a7c2570
Merge remote-tracking branch 'origin/s3-source-release' into polling_…
Jan 6, 2025
4dd9ad8
fixed PMD errors
Jan 6, 2025
f401478
fixed testing errors
Jan 6, 2025
1d73e7f
added javadoc
Jan 6, 2025
2c0426a
fixed backoff test issue
Jan 7, 2025
332c3f5
fixed testPollWithSlowProducer test issue
Jan 7, 2025
06a12a3
added test for AWSV2SourceClient rehydration
Jan 7, 2025
0edf114
Fix for slow polling test
Jan 7, 2025
da17e1f
Changes as per review
Jan 7, 2025
dc35ee4
Update commons/src/main/java/io/aiven/kafka/connect/common/source/Abs…
Claudenw Jan 9, 2025
7aa64fc
Update commons/src/main/java/io/aiven/kafka/connect/common/source/Abs…
Claudenw Jan 9, 2025
c25a877
Update commons/src/main/java/io/aiven/kafka/connect/common/source/Abs…
Claudenw Jan 9, 2025
37260b3
Update s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/so…
Claudenw Jan 9, 2025
21f713a
Update commons/src/main/java/io/aiven/kafka/connect/common/source/Abs…
Claudenw Jan 9, 2025
eb2c051
Update s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/so…
Claudenw Jan 9, 2025
a9d9380
Update s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/so…
Claudenw Jan 9, 2025
4f3f896
changes as per review requests
Jan 9, 2025
45bd6fd
fix for javadoc
Jan 9, 2025
08c3917
Merge remote-tracking branch 'refs/remotes/upstream/polling_efficienc…
Jan 9, 2025
523f1b6
fix for bad param
Jan 9, 2025
79c0063
fixed PMDerrors
Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions commons/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ dependencies {
testImplementation(jackson.databind)
testImplementation(testinglibs.mockito.core)
testImplementation(testinglibs.assertj.core)
testImplementation(testinglibs.awaitility)
testImplementation(testFixtures(project(":commons")))
testImplementation(testinglibs.woodstox.stax2.api)
testImplementation(apache.hadoop.mapreduce.client.core)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@

import io.aiven.kafka.connect.common.config.enums.ErrorsTolerance;
import io.aiven.kafka.connect.common.source.input.InputFormat;
import io.aiven.kafka.connect.common.source.input.Transformer;
import io.aiven.kafka.connect.common.source.input.TransformerFactory;

public class SourceCommonConfig extends CommonConfig {

Expand Down Expand Up @@ -64,11 +66,15 @@ public String getTargetTopicPartitions() {
}

public ErrorsTolerance getErrorsTolerance() {
return ErrorsTolerance.forName(sourceConfigFragment.getErrorsTolerance());
return sourceConfigFragment.getErrorsTolerance();
}

public int getMaxPollRecords() {
return sourceConfigFragment.getMaxPollRecords();
}

public Transformer getTransformer() {
return TransformerFactory.getTransformer(schemaRegistryFragment.getInputFormat());
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,8 @@ public int getExpectedMaxMessageBytes() {
return cfg.getInt(EXPECTED_MAX_MESSAGE_BYTES);
}

public String getErrorsTolerance() {
return cfg.getString(ERRORS_TOLERANCE);
public ErrorsTolerance getErrorsTolerance() {
return ErrorsTolerance.forName(cfg.getString(ERRORS_TOLERANCE));
}

private static class ErrorsToleranceValidator implements ConfigDef.Validator {
Expand Down
Loading
Loading