[Change Proposal] Streamline installation for ML integration packages

[jmcarlock]

Problem being solved

For DGA and Living off the Land, users currently need to go to the Dev console and manually assign custom component templates to data streams created by other integrations. This requires users to overwrite the default pipeline on a data stream.

Proposed changes

• Create a configuration UI/API to allow the user to select to apply a pipeline and field mappings to existing data streams (or allow users to assign one to their own custom data stream), similar to how is done with @custom component templates.
  • Potentially also add to the API to support arbitrary intermediate pipelines which can come between the default_pipeline and final_pipeline.
• The definition of supported data streams can be at the package level. For example DGA could have a configuration file indicating that if Elastic Defend is installed, add a checkbox for the user to apply DGA component template to logs-endpoint.events.network. In the UI, this could look like the configuration section for other Integration packages, like the Windows Integration (shown below).
  
• Possibly also link to the package's anomaly detectors/detection rules pages post installation.
• Updates should be as streamlined as possible, allowing users to easily update the integrations.
• Edit: Sept 6, 2024: As pointed out in the community slack, sometimes integration packages do not require being added to an agent policy, this could also be handled internally.
  

[pantea-elastic]

According to @Mikaayenson this might pose a challenge to TRADE's validations.