AutoTuner/Bootstrapper should recommend Dataproc Spark performance enhancements #1539
Conversation
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
parthosa
added
feature request
core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
There was a problem hiding this comment.
Thanks @parthosa !
nit: the test does not verify the behavior of the output in the following cases:
- The properties were already set by the user either true/false: just confirming that we want to override the value set by the user even when if they set it to false?
- The combined and the bootstrap output. It is fine to skip this since the implementation of those 2 parts is not completed anyway.
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Thanks for catching that. Added unit test for this.
Yes, we might need a framework to test these two files. Then, we could test all the recommendations. I think we should do that separately. |
| @@ -1117,8 +1117,9 @@ class AutoTuner( | |||
| calculateClusterLevelRecommendations() | |||
|
|
|||
| // add all platform specific recommendations | |||
| platform.recommendationsToInclude.foreach { | |||
| case (property, value) => appendRecommendation(property, value) | |||
| platform.recommendationsToInclude.collect { | |||
There was a problem hiding this comment.
This change prioritizes any user specified platform config over the ones coming from AutoTuner. We can update the comment to mention it if we plan to keeping this priority.
There was a problem hiding this comment.
Thanks @sayedbilalbari. Updated the PR description with the comment.
Fixes #1538.
This PR updates AutoTuner/Bootstrapper to recommend the following Dataproc Spark performance enhancements
Note:
Reference - https://cloud.google.com/dataproc/docs/guides/performance-enhancements