Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Disable dataproc enhanced optimizer configs #1550

Closed
amahussein opened this issue Feb 19, 2025 · 0 comments · Fixed by #1554
Closed

[BUG] Disable dataproc enhanced optimizer configs #1550

amahussein opened this issue Feb 19, 2025 · 0 comments · Fixed by #1554
Assignees
@amahussein
Labels
autotuner bug Something isn't working

Comments

@amahussein
Copy link
Collaborator

Describe the bug

After we merged #1539
GPU clusters had problems after setting those two properties.

spark.dataproc.enhanced.optimizer.enabled true
spark.dataproc.enhanced.execution.enabled true

We either need to disable those two properties or explicitly set them to false until we investigate them on GPUs.
@amahussein amahussein added autotuner bug Something isn't working labels Feb 19, 2025
@amahussein amahussein self-assigned this Feb 19, 2025
@amahussein amahussein mentioned this issue Feb 19, 2025
Open
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Feb 20, 2025
@amahussein
Disable dataproc enhanced optimizer configs
ca17528 
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>

Fixes NVIDIA#1550

For dataproc GPU clusters, the Autotuner should set the following 2
properties to false. In addition, the autotuner appends a comment to
warn the user that those properties might be problematic to the GPU run.

- "spark.dataproc.enhanced.optimizer.enabled": "false"
- "spark.dataproc.enhanced.execution.enabled": "false"

The autotuner output will add a message stating that:

"should be disabled. WARN: Turning this property on might case the GPU
accelerated Dataproc cluster to hang."
@amahussein amahussein mentioned this issue Feb 20, 2025
Merged
amahussein added a commit that referenced this issue Feb 21, 2025
@amahussein
Disable dataproc enhanced optimizer configs (#1554)
fc5facf 
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>

Fixes #1550

For dataproc GPU clusters, the Autotuner should set the following 2
properties to false. In addition, the autotuner appends a comment to
warn the user that those properties might be problematic to the GPU run.

- "spark.dataproc.enhanced.optimizer.enabled": "false"
- "spark.dataproc.enhanced.execution.enabled": "false"

The autotuner output will add a message stating that:

"should be disabled. WARN: Turning this property on might case the GPU
accelerated Dataproc cluster to hang."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
autotuner bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Disable dataproc enhanced optimizer configs amahussein/spark-rapids-tools
1 participant
@amahussein