Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make crawler more generic #158

Merged
merged 1 commit into from
Jan 8, 2024

Conversation

paullatzelsperger
Copy link
Member

@paullatzelsperger paullatzelsperger commented Dec 28, 2023

What this PR changes/adds

This PR makes the Federated Catalog a bit more generic by separating the Crawler from the Federated Catalog. This is done by the following work items:

  • introduce crawler-spi module: contains all interfaces related to the crawler
  • introduce crawler-core module: contains all core impls related to the crawler
  • various class renamings
    • rename CatalogCrawler -> Crawler
    • rename NodeQueryAdatper -> CrawlerAction
    • rename DspNodeQueryAdapter -> DspCatalogRequestAction
    • rename BatchedRequestFetcher -> BatchedCatalogRequestFetcher
  • generify update request/response
  • provide a RecurringExecutionPlan by default, and make the ExecutionPlan pluggable

Why it does that

To lay ground work for moving out the Crawler into either upstream EDC, or possibly a separate repository. That in turn will make the Crawler usable in other use cases as well, e.g.
crawling DID documents

Further notes

  • the end-to-end tests was refactored into using the JVM-based multi-runtime test instead of using docker-compose.
  • the CacheConfiguration class was simplified to be an interface containing constants, now called CacheSettings

Linked Issue(s)

Closes # <-- insert Issue number if one exists

@paullatzelsperger paullatzelsperger added enhancement New feature or request refactoring changes in structure, object hierarchy, renaming, etc. breaking-change Will require manual intervention for version update labels Dec 28, 2023
Copy link

github-actions bot commented Dec 28, 2023

Test Results

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 606636c. ± Comparison against base commit 8ea6ff4.

♻️ This comment has been updated with latest results.

@paullatzelsperger paullatzelsperger marked this pull request as ready for review December 28, 2023 13:08
@paullatzelsperger paullatzelsperger marked this pull request as draft December 28, 2023 13:12
@paullatzelsperger paullatzelsperger marked this pull request as ready for review December 28, 2023 13:34
@paullatzelsperger paullatzelsperger force-pushed the feat/make_crawler_more_generic branch from 435318e to 11b4965 Compare December 28, 2023 13:57
@paullatzelsperger paullatzelsperger force-pushed the feat/make_crawler_more_generic branch from 11b4965 to 606636c Compare December 28, 2023 14:00
Copy link

github-actions bot commented Jan 5, 2024

This pull request is stale because it has been open for 7 days with no activity.

Copy link
Contributor

@wolf4ood wolf4ood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@paullatzelsperger paullatzelsperger merged commit e95bdbc into main Jan 8, 2024
26 checks passed
@ndr-brt ndr-brt deleted the feat/make_crawler_more_generic branch January 8, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change Will require manual intervention for version update enhancement New feature or request refactoring changes in structure, object hierarchy, renaming, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants