Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
### What changes were proposed in this pull request? This PR proposes to disallow Spark Connect strings in `--master` when Spark API mode is `connect`. This is Python specific issue. ### Why are the changes needed? Should work as documented in #49107 ### Does this PR introduce _any_ user-facing change? Not yet because the main change has not been released (#49107) ### How was this patch tested? Manually tested: ``` ./bin/pyspark --master "sc://localhost:15002" --conf spark.api.mode=connect ``` ``` Python 3.11.9 (main, Apr 19 2024, 11:44:45) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information. /.../spark/python/pyspark/shell.py:77: UserWarning: Failed to initialize Spark session. warnings.warn("Failed to initialize Spark session.") Traceback (most recent call last): File "/.../spark/python/pyspark/shell.py", line 52, in <module> spark = SparkSession.builder.getOrCreate() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../spark/python/pyspark/sql/session.py", line 512, in getOrCreate raise PySparkRuntimeError( pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_INVALID] Master must either be yarn or start with spark, k8s, or local. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50000 from HyukjinKwon/SPARK-51254. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 6603a4e) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
- Loading branch information