Skip to content

Commit

Permalink
[SPARK-51192][CONNECT] Expose `processWithoutResponseObserverForTesti…
Browse files Browse the repository at this point in the history
…ng` in `SparkConnectPlanner`

### What changes were proposed in this pull request?

Adds a new `processWithoutResponseObserverForTesting` private-package method scoped to `planner`. This method is called by `SparkConnectPlannerTestUtils#transform`.

### Why are the changes needed?

#47816 attempted to move `MockObserver` into source code to address compilation errors when open-source libraries attempt to test their command plugin extensions via the `SparkConnectPlannerUtils`.

However, this isn't enough as the error `java.lang.NoSuchMethodError: 'void org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(org.apache.spark.connect.proto.Command, io.grpc.stub.StreamObserver` continues to be seen.

To address this shading issue, we move the creation of the `MockObserver` to the source code.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Existing tests

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49921 from vicennial/SPARK-51192.

Authored-by: vicennial <venkata.gudesa@databricks.com>
Signed-off-by: Herman van Hovell <herman@databricks.com>
  • Loading branch information
vicennial authored and hvanhovell committed Feb 17, 2025
1 parent 17b9431 commit 479e0b3
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -2474,6 +2474,15 @@ class SparkConnectPlanner(
}
}

/**
* Exposed for testing. Processes a command without a response observer.
*
* Called only from SparkConnectPlannerTestUtils.
*/
private[planner] def processWithoutResponseObserverForTesting(command: proto.Command): Unit = {
process(command, new MockObserver())
}

def process(
command: proto.Command,
responseObserver: StreamObserver[ExecutePlanResponse]): Unit = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ object SparkConnectPlannerTestUtils {

def transform(spark: SparkSession, command: proto.Command): Unit = {
val executeHolder = buildExecutePlanHolder(spark, command)
new SparkConnectPlanner(executeHolder).process(command, new MockObserver())
new SparkConnectPlanner(executeHolder).processWithoutResponseObserverForTesting(command)
}

private def buildExecutePlanHolder(
Expand Down

0 comments on commit 479e0b3

Please sign in to comment.