Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] AutoTuner/Bootstrapper should recommend Dataproc Spark performance enhancements configs #1538

Open
parthosa opened this issue Feb 7, 2025 · 1 comment · Fixed by #1539
Open

[FEA] AutoTuner/Bootstrapper should recommend Dataproc Spark performance enhancements configs #1538

parthosa opened this issue Feb 7, 2025 · 1 comment · Fixed by #1539
Assignees
@parthosa
Labels
autotuner feature request New feature or request

Comments

@parthosa
Copy link
Collaborator

parthosa commented Feb 7, 2025 •
edited
Loading

For Dataproc platform, AutoTuner/Bootstrapper should recommend Spark performance enhancements configs.

spark.dataproc.enhanced.optimizer.enabled=true
spark.dataproc.enhanced.execution.enabled=true

Reference - https://cloud.google.com/dataproc/docs/guides/performance-enhancements

@viadea @kuhushukla
@parthosa parthosa added ? - Needs Triage feature request New feature or request labels Feb 7, 2025
@parthosa parthosa self-assigned this Feb 7, 2025
@parthosa parthosa mentioned this issue Feb 7, 2025
Merged
@amahussein
Copy link
Collaborator

Reopening this as we faced problem with Accelerated GPU clusters after enabling those two configs.
We will disable them for now until we have more extensive testing #1550

@amahussein amahussein reopened this Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@parthosa parthosa

Labels
autotuner feature request New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@parthosa @amahussein