plugins: add clean command to remove inactive plugins/dependencies

[yaauie]

Release notes

• adds bin/logstash-plugin clean command to allow the plugin manager to prune orphaned dependencies

What does this PR do?

adds bin/logstash-plugin clean command to allow the plugin manager to prune orphaned dependencies

Why is it important/What is the impact to the user?

• For users who update plugins, providing a way to clean out deactivated dependencies can reduce the disk footprint of the logstash installation
• For building Logstash artifacts, a clean option allows us to build a true subset of the dependency graph after removing plugins that are not needed in a minimalized artifact

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works There are existing tests around LogStash::Bunder#invoke!(clean: true)

Author's Checklist

• File follow-up to document after the asciidoc-freeze on main is lifted.

How to test this PR locally

1. remove a plugin that has dependencies (like logstash-integration-aws)
   bin/logstash-plugin remove logstash-integration-aws
2. run the plugin manger's clean command
   bin/logstash-plugin clean
3. Observe that the orphaned plugin and dependencies are cleaned up and removed from vendor

Logs

╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-clean-command ✘) }
╰─● bin/logstash-plugin remove logstash-integration-aws
Using system java: /Users/rye/.jenv/shims/java
Resolving dependencies......
Resolving dependencies......
Successfully removed logstash-integration-aws
[success (00:00:06)]

╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-clean-command ✘) }
╰─● bin/logstash-plugin clean
Using system java: /Users/rye/.jenv/shims/java
cleaned inactive plugin logstash-integration-aws-7.1.8 (java)
cleaned inactive dependency aws-eventstream (1.3.0)
cleaned inactive dependency aws-partitions (1.1043.0)
cleaned inactive dependency aws-sdk-cloudfront (1.109.0)
cleaned inactive dependency aws-sdk-cloudwatch (1.109.0)
cleaned inactive dependency aws-sdk-core (3.217.0)
cleaned inactive dependency aws-sdk-kms (1.97.0)
cleaned inactive dependency aws-sdk-resourcegroups (1.77.0)
cleaned inactive dependency aws-sdk-s3 (1.179.0)
cleaned inactive dependency aws-sdk-sns (1.94.0)
cleaned inactive dependency aws-sdk-sqs (1.91.0)
cleaned inactive dependency aws-sigv4 (1.11.0)
cleaned inactive dependency jar-dependencies (0.4.1)
cleaned inactive dependency jmespath (1.6.2)
[success]

[donoghuc]

Seems pretty straight forward! I like partitioning by plugin gem vs supporting gem dep.

[donoghuc]

I'm a bit concerned about something... When i do a clean build (check out this code) and run ./gradlew --console=plain clean bootstrap assemble installDefaultGems to get a fresh build. Running the clean command a couple times removes some gems we actually need.... Once that happens I get messages about jar-dependencies not being able to be loaded etc. @yaauie Is this a problem with out I'm building/testing? Or is is this an issue we need to dig further in to?

➜  logstash git:(c4196a9676) ✗ bin/logstash-plugin clean
Using system java: /Users/cas/.jenv/shims/java
cleaned inactive dependency public_suffix (6.0.1)
cleaned inactive dependency ruby-maven (3.9.3)
➜  logstash git:(c4196a9676) ✗ bin/logstash-plugin clean
Using system java: /Users/cas/.jenv/shims/java
cleaned inactive dependency jar-dependencies (0.4.1)
➜  logstash git:(c4196a9676) ✗ bin/logstash-plugin clean
Using system java: /Users/cas/.jenv/shims/java

That said I did some additional manual testing with two example gems a and b:

➜  logstash git:(c4196a9676) ✗ cat a.gemspec
Gem::Specification.new do |s|
    s.name = 'a'
    s.version = '0.1.1'
    s.summary = "A depends on B"
    s.authors = ["Test"]
    s.files = [__FILE__]
    s.add_runtime_dependency "winrm", ">= 0.1"
    s.metadata = { "logstash_plugin" => "true" }
  end
➜  logstash git:(c4196a9676) ✗ cat b.gemspec
Gem::Specification.new do |s|
    s.name = 'b'
    s.version = '0.1.1'
    s.summary = "B - no dependencies"
    s.authors = ["Test"]
    s.files = [__FILE__]
    s.add_runtime_dependency "erubi", ">= 0"
    s.metadata = { "logstash_plugin" => "true" }
  end

I did combinations of adding/removing and cleaning and everything I could come up with worked as expected!

[yaauie]

Once that happens I get messages about jar-dependencies not being able to be loaded etc

That is in fact concerning. We're just proxying to bundler, so I don't see why that dependency should get cleaned. I'll dig in.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 28348fd

Failed CI Steps

• 🥼 Integration Tests / part 1

History

• 💛 Build #2355 was flaky 131e22c
• 💚 Build #2185 succeeded a9896fb