PPL `fieldsummary` command #766

YANG-DB · 2024-10-11T00:37:21Z

Description

This PR implements the fieldsummary PPL command.

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

- antlr syntax - ast expression builder - ast node builder - catalyst ast builder Signed-off-by: YANGDB <yang.db.dev@gmail.com>

fix scala style format Signed-off-by: YANGDB <yang.db.dev@gmail.com>

# Conflicts: # ppl-spark-integration/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java

…ng table identifier only has 2 parts) Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

# Conflicts: # ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin · 2024-10-18T08:53:07Z

Would you mind change the PR status to DRAFT if you are going to refactor?

YANG-DB · 2024-10-18T16:29:49Z

@LantaoJin plz review
I've removed the topValues in favour of performance - and added the next issue for enhancement of the top, rare commands

LantaoJin · 2024-10-22T01:57:09Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/NamedExpression.java

+@ToString
+@RequiredArgsConstructor
+@EqualsAndHashCode(callSuper = false)
+public class NamedExpression extends UnresolvedExpression {


Could you remove this definition or find an alternative, for example reusing Argument? Because

NamedExpression should have a name

NamedExpression generally should be an abstract class and the parent of Attribute or Alias. We should refactor many codes if we really need it.

It's confused me with Spark NamedExpression, it is not worth to introduce a new expression for fieldsummary command IMO. At least, not NamedExpression

LantaoJin · 2024-10-22T02:04:42Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/DataTypeTransformer.java

@@ -39,7 +42,8 @@
 */
 public interface DataTypeTransformer {
    static <T> Seq<T> seq(T... elements) {
-        return seq(List.of(elements));
+        return seq(Arrays.stream(elements).filter(Objects::nonNull)


Could you comment for this changes? What case did you see?

yes its not relevant any more - thanks for pointing it out

LantaoJin · 2024-10-22T02:09:31Z

...rk-integration/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java

  SUM(FunctionName.of("sum")),
  COUNT(FunctionName.of("count")),
+  COUNT_DISTINCT(FunctionName.of("count_distinct")),


We do have a DISTINCT_COUNT. (did we miss it here?) And seems you are not use it as a built-in function name in following codes, instead, count_distinct is an alias name.

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin · 2024-10-23T03:57:04Z

ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4

+FIELDSUMMARY:                       'FIELDSUMMARY';
+INCLUDEFIELDS:                      'INCLUDEFIELDS';
+NULLS:                              'NULLS';


We should add these keywords to keywordsCanBeId too.

LantaoJin · 2024-10-23T04:00:53Z

docs/ppl-lang/ppl-fieldsummary-command.md

+
+    os> source = t | fieldsummary includefields= id, status_code, request_path nulls=true
+    +------------------+-------------+------------+------------+------------+------------+------------+------------+----------------|
+    | Fiels            | COUNT       | COUNT_DISTINCT    |  MIN  |  MAX   |  AVG   |  MEAN   |        STDDEV       | NUlls | TYPEOF |


LantaoJin · 2024-10-23T04:02:16Z

docs/ppl-lang/ppl-fieldsummary-command.md

+
+    os> source = t | where status_code != 200 | fieldsummary includefields= status_code nulls=true
+    +------------------+-------------+------------+------------+------------+------------+------------+------------+----------------|
+    | Fiels            | COUNT       | COUNT_DISTINCT    |  MIN  |  MAX   |  AVG   |  MEAN   |        STDDEV       | NUlls | TYPEOF |


Fiels -> Fields?

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add support for FieldSummary - antlr syntax - ast expression builder - ast node builder - catalyst ast builder Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add support for FieldSummary - antlr syntax - ast expression builder - ast node builder - catalyst ast builder Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update sample query fix scala style format Signed-off-by: YANGDB <yang.db.dev@gmail.com> * support spark prior to 3.5 with its extended table identifier (existing table identifier only has 2 parts) Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update union queries based summary Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala fmt style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update scala fmt style Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update query with where clause predicate Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update command and remove the topvalues Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update command docs Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update with comments feedback Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update `FIELD SUMMARY` symbols to the keywordsCanBeId bag of words Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB added 4 commits October 9, 2024 19:51

add support for FieldSummary

cea6e44

- antlr syntax - ast expression builder - ast node builder - catalyst ast builder Signed-off-by: YANGDB <yang.db.dev@gmail.com>

add support for FieldSummary

fd1375a

- antlr syntax - ast expression builder - ast node builder - catalyst ast builder Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update sample query

4f30306

fix scala style format Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into ppl-fieldsummary-command

21924ae

# Conflicts: # ppl-spark-integration/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java

YANG-DB requested review from dai-chen, rupal-bq, vamsimanohar, penghuo, seankao-az, anirudha, kaituo and LantaoJin as code owners October 11, 2024 00:37

YANG-DB marked this pull request as draft October 11, 2024 00:37

YANG-DB added Lang:PPL Pipe Processing Language support 0.6 labels Oct 11, 2024

YANG-DB added 8 commits October 15, 2024 09:39

Merge branch 'main' into ppl-fieldsummary-command

a9e7c6e

support spark prior to 3.5 with its extended table identifier (existi…

7bcce2f

…ng table identifier only has 2 parts) Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update union queries based summary

c170208

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update scala fmt style

63c6118

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update scala fmt style

049be03

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update query with where clause predicate

ea5cbdf

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

update command and remove the topvalues

7a672fc

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into ppl-fieldsummary-command

c5c1cd3

# Conflicts: # ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

YANG-DB marked this pull request as ready for review October 17, 2024 22:15

update command docs

9236999

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Merge branch 'main' into ppl-fieldsummary-command

b483ff1

LantaoJin reviewed Oct 22, 2024

View reviewed changes

Merge branch 'main' into ppl-fieldsummary-command

ed5ccd9

update with comments feedback

132978b

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

LantaoJin reviewed Oct 23, 2024

View reviewed changes

YANG-DB requested a review from LantaoJin October 24, 2024 00:58

YANG-DB added 2 commits October 23, 2024 18:00

Merge branch 'main' into ppl-fieldsummary-command

94acd56

update FIELD SUMMARY symbols to the keywordsCanBeId bag of words

e3005d4

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

YANG-DB enabled auto-merge (squash) October 24, 2024 05:52

LantaoJin approved these changes Oct 24, 2024

View reviewed changes

seankao-az disabled auto-merge October 25, 2024 17:25

seankao-az merged commit 6513ead into opensearch-project:main Oct 25, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PPL `fieldsummary` command #766

PPL `fieldsummary` command #766

YANG-DB commented Oct 11, 2024

LantaoJin commented Oct 18, 2024

YANG-DB commented Oct 18, 2024

LantaoJin Oct 22, 2024

LantaoJin Oct 22, 2024

YANG-DB Oct 22, 2024

LantaoJin Oct 22, 2024

LantaoJin Oct 23, 2024

LantaoJin Oct 23, 2024

LantaoJin Oct 23, 2024

PPL fieldsummary command #766

PPL fieldsummary command #766

Conversation

YANG-DB commented Oct 11, 2024

Description

Issues Resolved

LantaoJin commented Oct 18, 2024

YANG-DB commented Oct 18, 2024

LantaoJin Oct 22, 2024

Choose a reason for hiding this comment

LantaoJin Oct 22, 2024

Choose a reason for hiding this comment

YANG-DB Oct 22, 2024

Choose a reason for hiding this comment

LantaoJin Oct 22, 2024

Choose a reason for hiding this comment

LantaoJin Oct 23, 2024

Choose a reason for hiding this comment

LantaoJin Oct 23, 2024

Choose a reason for hiding this comment

LantaoJin Oct 23, 2024

Choose a reason for hiding this comment

PPL `fieldsummary` command #766

PPL `fieldsummary` command #766