Skip to content

Commit

Permalink
[SPARK-51234][PYTHON][DOCS] Document an import change in `from pyspar…
Browse files Browse the repository at this point in the history
…k.sql.functions import *`

### What changes were proposed in this pull request?
Document an import change in `from pyspark.sql.functions import *`

### Why are the changes needed?
to notify users

### Does this PR introduce _any_ user-facing change?
doc-only

### How was this patch tested?
ci

### Was this patch authored or co-authored using generative AI tooling?
no

Closes #49975 from zhengruifeng/py_funcs_wildcard_import.

Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
  • Loading branch information
zhengruifeng committed Feb 18, 2025
1 parent aeea738 commit 2c76dff
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions python/docs/source/migration_guide/pyspark_upgrade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, the schema of a map column is inferred by merging the schemas of all pairs in the map. To restore the previous behavior where the schema is only inferred from the first non-null pair, you can set ``spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled`` to ``true``.
* In Spark 4.0, `compute.ops_on_diff_frames` is on by default. To restore the previous behavior, set `compute.ops_on_diff_frames` to `false`.
* In Spark 4.0, the data type `YearMonthIntervalType` in ``DataFrame.collect`` no longer returns the underlying integers. To restore the previous behavior, set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``).


Upgrading from PySpark 3.3 to 3.4
Expand Down

0 comments on commit 2c76dff

Please sign in to comment.