Skip to content

[test] Fix flakey dvc delayed ingestion and killed repush status test #1788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

misyel
Copy link
Contributor

@misyel misyel commented May 12, 2025

Problem Statement

The tests for dvc delayed ingestion are flakey because the push status store may not be ready yet when we start the dvc client. And the test for the killed repush job status is flakey because the admin message isn't always consumed right away and we need to wait for it to be consumed.

Solution

  1. For dvc delayed ingestion tests, add a check to wait for the push status store to be ready before starting the dvc client
  2. For the killed repush job test, add in a check that waits for the KILLED version status that happens after the admin message is consumed

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.

@@ -793,4 +798,17 @@ public void testDvcDelayedIngestionWithFailingPushInTargetRegion() throws Except

client1.close();
}

public void verifyThatPushStatusStoreIsOnline(String storeName) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why public ? can you move it to a test utils class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a mistake, I updated it to private

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants