Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Refactor AutoTuner to work on CPU and GPU profiles #700

Closed
2 tasks done
Tracked by #697
amahussein opened this issue Dec 19, 2023 · 0 comments · Fixed by #739
Closed
2 tasks done
Tracked by #697

[FEA] Refactor AutoTuner to work on CPU and GPU profiles #700

amahussein opened this issue Dec 19, 2023 · 0 comments · Fixed by #739
Assignees
@amahussein
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@amahussein
Copy link
Collaborator

amahussein commented Dec 19, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.

Currently, AutoTuner handles the GPU profiles.
In order to automatically map CPU configs to GPU configurations, we want to reuse the logic that is already defined in the AutoTuner.

Describe the solution you'd like

Refactor AutoTuner to support the logic for both Profiler/Qualification

Describe alternatives you've considered

Rewrite a separate Spark configuration mapper implies we have duplicate logic for tunining the configurations.
That's the main reason we should do it

Tasks
Preview Give feedback
  1. [FEA] AutoTuner should not skip non-gpu eventlogs #726
    core_tools feature request
    amahussein
  2. [FEA] Collect App properties by the Qualification tool #733
    core_tools feature request
    amahussein
@amahussein amahussein added feature request New feature or request ? - Needs Triage core_tools Scope the core module (scala) labels Dec 19, 2023
@amahussein amahussein self-assigned this Dec 19, 2023
@amahussein amahussein mentioned this issue Dec 19, 2023
Closed
@amahussein amahussein mentioned this issue Jan 18, 2024
Closed
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Jan 24, 2024
@amahussein
[FEA] Qualification tool triggers the AtutoTuner module
2a01846 
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>

Fixes NVIDIA#700

This is an incremental step toward the full automation of App migration to GPU.

- Add Qual arg `--auto-tuner` to toggle the AutoTuner module. Default is Off.
- Add Qual arg `--worker-info` to pass the GPU worker info to the Qual's AutoTuner.
- When AutoTuner is enabled, the Qual tool will launch the AutoTuner module to make some basic recommendations/comments based on the Spark/Env properties.
- A new folder `rapids_4_spark_qualification_output/tuning` is created which contains a text formatted file for each app. Each file is named after the AppID.
- No unit-tests is added for now because: 1- the recommendations are based on the Profiler's implementation; and the feature is disabled by default.
- There will be followup to incrementally split the logic of the AutoTuner into two classes that aim to tailor the rules/policies of the recommendations to the CPU applications.
@amahussein amahussein mentioned this issue Jan 24, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[FEA] Qualification tool triggers the AutoTuner module amahussein/spark-rapids-tools
1 participant
@amahussein