@@ -41,7 +41,7 @@ class SparkKoalasGraphAdapter(base.HamiltonGraphAdapter, base.ResultMixin):
41
41
using the \
42
42
`Pandas API on Spark <https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html>`__
43
43
44
- Use `pip install sf-hamilton[spark]` to get the dependencies required to run this.
44
+ Use `` pip install sf-hamilton[spark]` ` to get the dependencies required to run this.
45
45
46
46
Currently, this class assumes you're running SPARK 3.2+. You'd generally use this if you have an existing spark \
47
47
cluster running in your workplace, and you want to scale to very large data set sizes.
@@ -759,7 +759,7 @@ def transform_node(
759
759
760
760
Note that, at this point, we don't actually know which columns will come from the
761
761
base dataframe, and which will come from the upstream nodes. This is handled in the
762
- `with_columns` decorator, so for now, we need to give it enough information to topologically
762
+ `` with_columns` ` decorator, so for now, we need to give it enough information to topologically
763
763
sort/assign dependencies.
764
764
765
765
:param node_: Node to transform
@@ -950,10 +950,10 @@ def __init__(
950
950
"""Initializes a with_columns decorator for spark. This allows you to efficiently run
951
951
groups of map operations on a dataframe, represented as pandas/primitives UDFs. This
952
952
effectively "linearizes" compute -- meaning that a DAG of map operations can be run
953
- as a set of `.withColumn` operations on a single dataframe -- ensuring that you don't have
954
- to do a complex `extract` then `join` process on spark, which can be inefficient.
953
+ as a set of `` .withColumn` ` operations on a single dataframe -- ensuring that you don't have
954
+ to do a complex `` extract`` then `` join` ` process on spark, which can be inefficient.
955
955
956
- Here's an example of calling it -- if you've seen `@subdag`, you should be familiar with
956
+ Here's an example of calling it -- if you've seen :py:class: `@subdag <hamilton.function_modifiers.recursive> `, you should be familiar with
957
957
the concepts:
958
958
959
959
.. code-block:: python
0 commit comments