Skip to content

Commit

Permalink
[SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR proposes to disallow Spark Connect strings in `--master` when Spark API mode is `connect`. This is Python specific issue.

### Why are the changes needed?

Should work as documented in #49107

### Does this PR introduce _any_ user-facing change?

Not yet because the main change has not been released (#49107)

### How was this patch tested?

Manually tested:

```
 ./bin/pyspark --master "sc://localhost:15002" --conf spark.api.mode=connect
```
```
Python 3.11.9 (main, Apr 19 2024, 11:44:45) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
/.../spark/python/pyspark/shell.py:77: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../spark/python/pyspark/shell.py", line 52, in <module>
    spark = SparkSession.builder.getOrCreate()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../spark/python/pyspark/sql/session.py", line 512, in getOrCreate
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_INVALID] Master must either be yarn or start with spark, k8s, or local.
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #50000 from HyukjinKwon/SPARK-51254.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 6603a4e)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon committed Feb 20, 2025
1 parent aa3c1e2 commit b9c96c7
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 0 deletions.
5 changes: 5 additions & 0 deletions python/pyspark/errors/error-conditions.json
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,11 @@
"Variant binary is malformed. Please check the data source is valid."
]
},
"MASTER_URL_INVALID": {
"message": [
"Master must either be yarn or start with spark, k8s, or local."
]
},
"MASTER_URL_NOT_SET": {
"message": [
"A master URL must be set in your configuration."
Expand Down
5 changes: 5 additions & 0 deletions python/pyspark/sql/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,11 @@ def getOrCreate(self) -> "SparkSession":

if url is None and is_api_mode_connect:
url = opts.get("spark.master", os.environ.get("MASTER", "local"))
if url.startswith("sc://"):
raise PySparkRuntimeError(
errorClass="MASTER_URL_INVALID",
messageParameters={},
)

if url is None:
raise PySparkRuntimeError(
Expand Down

0 comments on commit b9c96c7

Please sign in to comment.