-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support OpenSearch alias field type #1032
Conversation
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes!
flint-spark-integration/src/main/scala/org/apache/spark/sql/flint/json/FlintJacksonParser.scala
Show resolved
Hide resolved
LGTM! One minor concern from me is that the implementation of this PR is more like the Option3 we mentioned in: opensearch-project/sql#3246 (comment) So there are some minor behavioral differences between OpenSearch index PPL and Flint PPL. Since we populate the InternalRow with alias columns filled with real data during scanning, the alias columns won't reflect updates made to the original columns in the following execution. Nevertheless, since Spark doesn't support alias types, implementing like Option1 is challenging in Flint. Besides, it's a very edge case where users would query using alias columns while updating the original columns in this query. |
Yes, it is option3. |
Description
Summary
This PR introduces support for alias fields in
FlintDataType
, allowing fields to be referenced by alternative names. The change enables schema parsing and JSON processing to recognize alias fields and map them to their corresponding primary fields.Changes
FlintDataType.scala
deserializeJValue
to identify and process alias fields.deserializeField
to support alias metadata assignment.FlintJacksonParser.scala
fieldMapping
to associate JSON keys with schema field indices, allowing alias fields to be correctly resolved during parsing.Add OpenSearch Table datatype in docs
Testing
Related Issues
#1033
Check List
--signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.