[BUG] OSD integTest lack additional_config entrypoint for both OS and OSD clusters

[peterzhuamazon]

Hi,

This is related to OSD specific integTest.

In OS, the additional config entry can be grab from the opensearch-<>-test.json and added to the opensearch.yml.

In OSD, this functional is half-complete in service_opensearch_dashboards.py and only allow hardcoded entries, without interpreting the entries from opensearch-dashboards-<>-test.json.

Moreover, in OSD integTest you need to start both OS and OSD, means the additional config in OS also need a new param to hold, so it wont get mixed up with OSD additional config. Right now both are under additional-cluster-configs key and will both be pushed to opensearch_dashboards.yml.

Thank.

[peterzhuamazon]

Only partially resolved this issue in PR:

• Add additional cluster config support in OSD integTest #3590
• Initial empty dictionary for None additional cluster configs in OSD IntegTest #3593

[gaiksaya]

[Triage] The issue is partially resolved.

[peterzhuamazon]

We have implemented the OSD side of configs already.
As for OS configs during OSD test there does not seems to have a specific need from plugins.
Will close this issue for now until further notices. Thanks.

[peterzhuamazon]

Seems like there is a use case here:

• feat: remove useless config #4402

[kavilla]

Low priority: for OpenSearch Dashboards we set cluster.routing.allocation.disk.threshold_enabled=false in our GitHub workflows. It would be a nice-to-have feature to be configure OpenSearch when we needed

Priority is low because we should figure out why that threshold is being hit.

[ashwin-pc]

I wouldn't categorize this as a low priority since this is blocking running the tests with the same environment today locally and in the infra pipeline. Having identical test environments is key to reproducibility. Figuring out why we have to increase the memory limits is a different issue and I don't think we should tie the two together since that may be a bigger problem to solve and we shouldn't hold off on this feature on the hopes that that will be resolved.