[FEA] Qualification tool can map spark configs for GPU migration

[amahussein]

Is your feature request related to a problem? Please describe.

The Qualification tool currently shows a list of recommendations to bootstrap a GPU cluster.

For developers, this implies to reconfigure their workload by merging the recommended GPU ones with the original configs.
This usually lead to less efficient configurations for the workload.

In order to make it easier for the developers, the qualification tool can automatically generate the new set of configurations, along with comments to clarify the changes that were done.

We can do that in the following steps:

1. the Qualification tool marks the obsolete configs. This requires we enumerate the configurations that do not work for GPU workload and recommend that the user removes them.
   • Each added recommendation should have a comment explaining the reason behind it
   • Each removed configuration should have explanation too.
2. next step is to process the remainder of the configurations and making recommendations to update or append new lines as necessary.

Expected changes:

• Qualification should be changed to summarize the spark configs
• The output files/logs should include the version number of the tools to be able to know whether the recommendations were outdated or not.
• The Qualification needs to infer the cluster shape ([FEA] Qualification tool can infer the CPU jobs' cluster shape and then provide the suggestion based on that #581)

Tasks

Preview Give feedback

1. [FEA] Generate an output file with runtime information #699
   core_tools feature request +2
   
2. [FEA] Refactor AutoTuner to work on CPU and GPU profiles #700
   2 of 2
   core_tools feature request +2
   
3. [FEA] Qualification user tools runs AutoTuner by default #744
   feature request user_tools +2
   
4. [FEA] Redact SparkProperties in AutoTuner and Profiler outputs #772
   core_tools feature request +2
   

[amahussein]

completed by #771