[RFC] Handling OpenSearch geo_point Data in Spark

[ykmr1224]

1. Introduction

1.1 Background

When using OpenSearch as a data source for Apache Spark, special handling is required for OpenSearch-specific data types, such as geo_point. Spark does not have a native equivalent for geo_point, so a standard approach must be established for efficient querying and geospatial analysis.
Related Issue: #1047

1.2 Objective

This RFC proposes an approach for integrating OpenSearch geo_point fields into Spark DataFrames, ensuring proper data representation and efficient querying without relying on external geospatial libraries.

2. Problem Statement

2.1 OpenSearch geo_point Representation

OpenSearch stores geo_point fields in multiple formats, including:

• Array format: [longitude, latitude]
• String format: "lat,lon" or "POINT(longitude latitude)"
• Object format: { "lat": value, "lon": value }
  (ref: https://opensearch.org/docs/latest/field-types/supported-field-types/geo-point/)

Spark does not natively support geo_point, so an appropriate transformation is needed when retrieving data.

2.2 Challenges

• Consistency: OpenSearch allows multiple formats for geo_point, making data retrieval non-trivial.
• Query Compatibility: Efficient querying in Spark requires a structured geospatial format.
• Performance Considerations: Converting and processing geo_point fields efficiently is critical for large datasets.

3. Proposed Solution

3.1 Mapping OpenSearch geo_point to Spark DataFrames

We propose normalizing geo_point fields into one of the following Spark representations:

OpenSearch Format	Proposed Spark Representation
[lon, lat] (Array)	STRUCT<lon: DOUBLE, lat: DOUBLE>
{"lat": x, "lon": y} (Object)	STRUCT<lon: DOUBLE, lat: DOUBLE>
"lat,lon" (String)	STRUCT<lon: DOUBLE, lat: DOUBLE>
"POINT(lon lat)" (WKT)	STRING (WKT format)

3. Alternatives Considered

3.1 Keeping Original Representation

• Pros: Retains full fidelity of OpenSearch data without transformation. Output will remain the same as original data format.
• Cons: Requires additional processing in Spark for querying and analysis, leading to potential inefficiencies.
• Use Case: If we push down all of geospatial operations to OpenSearch, this approach might be preferable.

3.2 Keeping geo_point as JSON String

• Pros: Requires minimal transformation.
• Cons: Harder to use in queries; requires custom parsing.

3.3 Using WKT Representation

• Pros: Compatible with SQL-based geospatial processing.
• Cons: Requires conversion before analysis, increasing processing overhead.

3.4 Using UDT

• Pros: Potentially reducing memory footprint and I/O overhead due to custom serialization
• Cons: Requires additional conversions to make it operable with other data types. Performance overhead to serialize/deserialize to/from build-in type.

4. Open Questions

1. Should we support multiple representations (e.g., WKT and Struct) or standardize on one?
2. What optimizations should be applied when filtering OpenSearch geo_point data?

Discussion & Feedback

Please provide feedback by commenting on this RFC or suggesting changes via a pull request.

[penghuo]

Did u consider map to geo_point (UDT) type in spark?

[ykmr1224]

Did u consider map to geo_point (UDT) type in spark?

Added that option to 3.4. I did not see much benefit to make it UDT.

[LantaoJin]

High-level question: is this target to unified the data type for PPL language or we see some requirements from reality world (e.g. CWL)?

[LantaoJin]

Maybe we could learn from Apache Sedona (https://github.com/apache/sedona) which provides spatial computing based on Apache Spark.

[ykmr1224]

High-level question: is this target to unified the data type for PPL language or we see some requirements from reality world (e.g. CWL)?

We don't have specific requirements, yet. This is part of supporting OpenSearch sample data set (They include geo_point fields).

[ykmr1224]

Maybe we could learn from Apache Sedona (https://github.com/apache/sedona) which provides spatial computing based on Apache Spark.

That is very good point. Apache Sedona utilize UDT and store geometry data as Geometry class (serialized to Array[Byte]) provided by JTS Topology Suite. And it provides lots of UDFs (ref) to allow geographical calculation, aggregation, etc.
If we want to provide similar functionalities, UDT would be better approach since it would be easier to handle the data in each UDF as UTD can provide type safety and common functionality to UDFs. But from the fact that we don't have geo spacial functions in SQL plugin, I don't expect we implement many UDFs and then we cannot get benefit from this.