Expand ppl command #868

YANG-DB · 2024-11-05T00:28:05Z

Description

support expand command

Related Issues

Check List

Updated documentation (docs/ppl-lang/README.md)
Implemented unit tests
Implemented tests for combination with other commands
New added source code should include a copyright header
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

# Conflicts: # ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4 # ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

# Conflicts: # ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4 # ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin · 2024-11-05T07:39:59Z

docs/ppl-lang/PPL-Example-Commands.md

+#### **expand**
+[See additional command details](ppl-expand-command.md)
+```sql
+   -  `source=  table | expand field_with_array as array_list`


Could we add this example here:

source= table | expand json_array(1, 2, 3) as uid | fields uid (returns 3 rows with values 1, 2 and 3)

And add a similar test case in IT.

@LantaoJin question:
currently both flatten & expand only support fieldExpression

expandCommand : EXPAND fieldExpression (AS alias = qualifiedName)? ; flattenCommand : FLATTEN fieldExpression ;

Maybe using the next slightly different version of this expand array query ?
source = table | eval array=json_array(1, 2, 3) | expand array as uid | fields name, occupation, uid
would give a similar functional result without actually changing the grammar ?

LantaoJin · 2024-11-05T07:51:24Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

+        Optional<Expression> alias = node.getAlias().map(aliasNode -> visitExpression(aliasNode, context));
+        context.retainAllNamedParseExpressions(p -> (NamedExpression) p);
+        Explode explodeGenerator = new Explode(field);
+        scala.collection.mutable.Seq seq = alias.isEmpty() ? seq() : seq(alias.get());


Could you rename the variable seq to outputs? It is confusing.

LantaoJin · 2024-11-05T08:05:35Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

+        Explode explodeGenerator = new Explode(field);
+        scala.collection.mutable.Seq seq = alias.isEmpty() ? seq() : seq(alias.get());
+        if(alias.isEmpty())
+            return context.apply(p -> new Generate(new GeneratorOuter(explodeGenerator), seq(), true, (Option) None$.MODULE$, seq, p));


IMO, no need the GeneratorOuter here. How about simplify to

return context.apply(p -> new Generate(explodeGenerator, seq(), false, (Option) None$.MODULE$, seq, p));

Note. the third parameter outer should be false either.

@LantaoJin the issue here is that in the case where row is null not using GeneratorOuter will not output the row as shown here:

( 1, array(STRUCT("1_one", 1), STRUCT(null, 11), STRUCT("1_three", null)) ), ( 2, array(STRUCT("2_Monday", 2), null) ), ( 3, array(STRUCT("3_third", 3), STRUCT("3_4th", 4)) ), ( 4, null )

For the query:
source = $multiValueTable | expand multi_value AS exploded_multi_value | fields exploded_multi_value

LantaoJin · 2024-11-05T08:06:51Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

+            return context.apply(p -> new Generate(new GeneratorOuter(explodeGenerator), seq(), true, (Option) None$.MODULE$, seq, p));
+        else {
+            //in case an alias does appear - remove the original field from the returning columns
+            context.apply(p -> new Generate(new GeneratorOuter(explodeGenerator), seq(), true, (Option) None$.MODULE$, seq, p));


remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add expand command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add expand command with visitor Signed-off-by: YANGDB <yang.db.dev@gmail.com> * create unit / integration tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update expand tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update doc Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update docs with examples Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update with additional test case remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update with additional test case remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update documentation Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com> (cherry picked from commit 4303057) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* add expand command * add expand command with visitor * create unit / integration tests * update expand tests * add tests * update doc * update docs with examples * update scala style * update with additional test case remove outer generator * update with additional test case remove outer generator * update documentation --------- (cherry picked from commit 4303057) Signed-off-by: YANGDB <yang.db.dev@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* add expand command Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add expand command with visitor Signed-off-by: YANGDB <yang.db.dev@gmail.com> * create unit / integration tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update expand tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add tests Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update doc Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update docs with examples Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update with additional test case remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update with additional test case remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update documentation Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB added 8 commits October 31, 2024 13:23

add expand command

35ea4d1

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into expand-ppl-command

54abdbe

# Conflicts: # ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4 # ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4

add expand command with visitor

a6e76ec

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

create unit / integration tests

f0335cc

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into expand-ppl-command

a54262e

# Conflicts: # ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4 # ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

update expand tests

d8a003f

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into expand-ppl-command

7b50ffa

add tests

71b7497

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB requested review from dai-chen, rupal-bq, mengweieric, vamsimanohar, penghuo, seankao-az, anirudha, kaituo, noCharger, LantaoJin and ykmr1224 as code owners November 5, 2024 00:28

YANG-DB marked this pull request as draft November 5, 2024 00:28

YANG-DB added 2 commits November 4, 2024 16:41

update doc

ef1d890

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update docs with examples

3db02ce

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB added Lang:PPL Pipe Processing Language support 0.6 labels Nov 5, 2024

update scala style

76baeee

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin reviewed Nov 5, 2024

View reviewed changes

Merge branch 'main' into expand-ppl-command

ff39d31

YANG-DB marked this pull request as ready for review November 5, 2024 22:20

YANG-DB added 2 commits November 5, 2024 14:51

update with additional test case

a91a986

remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update with additional test case

82f7135

remove outer generator Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB requested a review from LantaoJin November 6, 2024 18:14

update documentation

cdadb18

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin approved these changes Nov 7, 2024

View reviewed changes

LantaoJin merged commit 4303057 into opensearch-project:main Nov 7, 2024
4 checks passed

anasalkouz added the backport 0.6 label Nov 12, 2024

opensearch-trigger-bot bot mentioned this pull request Nov 12, 2024

[Backport 0.6] Expand ppl command #895

Merged

qianheng-aws mentioned this pull request Nov 15, 2024

[BUG] Expand doesn't support alias in GA environment #910

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand ppl command #868

Expand ppl command #868

YANG-DB commented Nov 5, 2024

LantaoJin Nov 5, 2024

LantaoJin Nov 5, 2024

YANG-DB Nov 5, 2024 •

edited

Loading

LantaoJin Nov 5, 2024

LantaoJin Nov 5, 2024 •

edited

Loading

YANG-DB Nov 5, 2024

LantaoJin Nov 5, 2024

Expand ppl command #868

Expand ppl command #868

Conversation

YANG-DB commented Nov 5, 2024

Description

Related Issues

Check List

LantaoJin Nov 5, 2024

Choose a reason for hiding this comment

LantaoJin Nov 5, 2024

Choose a reason for hiding this comment

YANG-DB Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

LantaoJin Nov 5, 2024

Choose a reason for hiding this comment

LantaoJin Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

YANG-DB Nov 5, 2024

Choose a reason for hiding this comment

LantaoJin Nov 5, 2024

Choose a reason for hiding this comment

YANG-DB Nov 5, 2024 •

edited

Loading

LantaoJin Nov 5, 2024 •

edited

Loading