-
Notifications
You must be signed in to change notification settings - Fork 42
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Disable Per-SQL summary text output (#1530)
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me> Fixes #1527 Disable the text format output generated per-sql. The target is to reduce the noise of the stdout and improve the performance of the core-tools ### Impact on the output: - `rapids_4_spark_qualification_output_persql.log` is not generated anymore by the qualTool - remove column `AppName` from `rapids_4_spark_qualification_output_persql.csv` - the `rapids_4_spark_qualification_output_persql.log` can still be generated by the RunningQualificationApp - The order of the SQls in the CSV file has changed. Sorted Desc based on (GPU opportunity, and DF duration) whithin each app. Perviously, the SQLs were sorted globally which might cause considerable overhead for a large number of eventogs. ### Impact on Performance and usability: - Improve readability of the stdout/log generated by the tools. - Reduce the size of lines consumed by the python wrapper. - Sorting Sqls per-app implies less memory requirements since it is only required to maintain the list of SQL for the current iteration. - Improve the string construction by avoiding filling `Buffer<String, Int>` - Improve the performance by skipping generating and writing the log file to the disk.
- Loading branch information
1 parent
c8fedd7
commit 38aad12
Showing
8 changed files
with
172 additions
and
123 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
52 changes: 26 additions & 26 deletions
52
core/src/test/resources/QualificationExpectations/nds_q86_fail_test_expectation_persql.csv
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,26 +1,26 @@ | ||
App Name,App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",0,"Register input tables",2,2 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",21,"Register input tables",1,1 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",5,"Register input tables",1,1 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",6,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",15,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",3,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",12,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",18,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",9,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",19,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",1,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",10,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",16,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",7,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",22,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",13,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",4,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",14,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",20,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",2,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",11,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",17,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",8,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",23,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",24,"Benchmark Run: query=q86; iteration=0",9565,9565 | ||
App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"app-20210319163812-1778","",0,"Register input tables",2,2 | ||
"app-20210319163812-1778","",21,"Register input tables",1,1 | ||
"app-20210319163812-1778","",5,"Register input tables",1,1 | ||
"app-20210319163812-1778","",6,"Register input tables",0,0 | ||
"app-20210319163812-1778","",15,"Register input tables",0,0 | ||
"app-20210319163812-1778","",3,"Register input tables",0,0 | ||
"app-20210319163812-1778","",12,"Register input tables",0,0 | ||
"app-20210319163812-1778","",18,"Register input tables",0,0 | ||
"app-20210319163812-1778","",9,"Register input tables",0,0 | ||
"app-20210319163812-1778","",19,"Register input tables",0,0 | ||
"app-20210319163812-1778","",1,"Register input tables",0,0 | ||
"app-20210319163812-1778","",10,"Register input tables",0,0 | ||
"app-20210319163812-1778","",16,"Register input tables",0,0 | ||
"app-20210319163812-1778","",7,"Register input tables",0,0 | ||
"app-20210319163812-1778","",22,"Register input tables",0,0 | ||
"app-20210319163812-1778","",13,"Register input tables",0,0 | ||
"app-20210319163812-1778","",4,"Register input tables",0,0 | ||
"app-20210319163812-1778","",14,"Register input tables",0,0 | ||
"app-20210319163812-1778","",20,"Register input tables",0,0 | ||
"app-20210319163812-1778","",2,"Register input tables",0,0 | ||
"app-20210319163812-1778","",11,"Register input tables",0,0 | ||
"app-20210319163812-1778","",17,"Register input tables",0,0 | ||
"app-20210319163812-1778","",8,"Register input tables",0,0 | ||
"app-20210319163812-1778","",23,"Register input tables",0,0 | ||
"app-20210319163812-1778","",24,"Benchmark Run: query=q86; iteration=0",9565,9565 |
52 changes: 26 additions & 26 deletions
52
core/src/test/resources/QualificationExpectations/nds_q86_test_expectation_persql.csv
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,26 +1,26 @@ | ||
App Name,App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",24,"Benchmark Run: query=q86; iteration=0",9565,9565 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",0,"Register input tables",2,2 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",21,"Register input tables",1,1 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",5,"Register input tables",1,1 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",6,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",15,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",3,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",12,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",18,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",9,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",19,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",1,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",10,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",16,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",7,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",22,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",13,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",4,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",14,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",20,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",2,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",11,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",17,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",8,"Register input tables",0,0 | ||
"TPC-DS Like Bench q86","app-20210319163812-1778","",23,"Register input tables",0,0 | ||
App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"app-20210319163812-1778","",24,"Benchmark Run: query=q86; iteration=0",9565,9565 | ||
"app-20210319163812-1778","",0,"Register input tables",2,2 | ||
"app-20210319163812-1778","",21,"Register input tables",1,1 | ||
"app-20210319163812-1778","",5,"Register input tables",1,1 | ||
"app-20210319163812-1778","",6,"Register input tables",0,0 | ||
"app-20210319163812-1778","",15,"Register input tables",0,0 | ||
"app-20210319163812-1778","",3,"Register input tables",0,0 | ||
"app-20210319163812-1778","",12,"Register input tables",0,0 | ||
"app-20210319163812-1778","",18,"Register input tables",0,0 | ||
"app-20210319163812-1778","",9,"Register input tables",0,0 | ||
"app-20210319163812-1778","",19,"Register input tables",0,0 | ||
"app-20210319163812-1778","",1,"Register input tables",0,0 | ||
"app-20210319163812-1778","",10,"Register input tables",0,0 | ||
"app-20210319163812-1778","",16,"Register input tables",0,0 | ||
"app-20210319163812-1778","",7,"Register input tables",0,0 | ||
"app-20210319163812-1778","",22,"Register input tables",0,0 | ||
"app-20210319163812-1778","",13,"Register input tables",0,0 | ||
"app-20210319163812-1778","",4,"Register input tables",0,0 | ||
"app-20210319163812-1778","",14,"Register input tables",0,0 | ||
"app-20210319163812-1778","",20,"Register input tables",0,0 | ||
"app-20210319163812-1778","",2,"Register input tables",0,0 | ||
"app-20210319163812-1778","",11,"Register input tables",0,0 | ||
"app-20210319163812-1778","",17,"Register input tables",0,0 | ||
"app-20210319163812-1778","",8,"Register input tables",0,0 | ||
"app-20210319163812-1778","",23,"Register input tables",0,0 |
36 changes: 18 additions & 18 deletions
36
core/src/test/resources/QualificationExpectations/qual_test_simple_expectation_persql.csv
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,18 @@ | ||
App Name,App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",1,"count at QualificationInfoUtils.scala:94",7143,6719 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",3,"count at QualificationInfoUtils.scala:94",2052,1660 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",2,"count at QualificationInfoUtils.scala:94",1933,1551 | ||
"Spark shell","local-1651187225439","",0,"show at <console>:26",498,333 | ||
"Spark shell","local-1651188809790","",0,"show at <console>:26",715,242 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",0,"json at QualificationInfoUtils.scala:76",1306,164 | ||
"Spark shell","local-1651188809790","",1,"show at <console>:26",196,135 | ||
"Spark shell","local-1651187225439","",1,"show at <console>:26",262,110 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",2,"json at QualificationInfoUtils.scala:136",321,107 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",5,"json at QualificationInfoUtils.scala:136",129,43 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",8,"json at QualificationInfoUtils.scala:136",127,42 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",4,"createOrReplaceTempView at QualificationInfoUtils.scala:133",22,22 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",7,"createOrReplaceTempView at QualificationInfoUtils.scala:133",4,4 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",1,"createOrReplaceTempView at QualificationInfoUtils.scala:133",2,2 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",0,"json at QualificationInfoUtils.scala:130",1209,0 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",6,"json at QualificationInfoUtils.scala:130",110,0 | ||
"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",3,"json at QualificationInfoUtils.scala:130",108,0 | ||
App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportunity | ||
"local-1622043423018","",1,"count at QualificationInfoUtils.scala:94",7143,6719 | ||
"local-1622043423018","",3,"count at QualificationInfoUtils.scala:94",2052,1660 | ||
"local-1622043423018","",2,"count at QualificationInfoUtils.scala:94",1933,1551 | ||
"local-1651187225439","",0,"show at <console>:26",498,333 | ||
"local-1651188809790","",0,"show at <console>:26",715,242 | ||
"local-1622043423018","",0,"json at QualificationInfoUtils.scala:76",1306,164 | ||
"local-1651188809790","",1,"show at <console>:26",196,135 | ||
"local-1651187225439","",1,"show at <console>:26",262,110 | ||
"local-1623281204390","",2,"json at QualificationInfoUtils.scala:136",321,107 | ||
"local-1623281204390","",5,"json at QualificationInfoUtils.scala:136",129,43 | ||
"local-1623281204390","",8,"json at QualificationInfoUtils.scala:136",127,42 | ||
"local-1623281204390","",4,"createOrReplaceTempView at QualificationInfoUtils.scala:133",22,22 | ||
"local-1623281204390","",7,"createOrReplaceTempView at QualificationInfoUtils.scala:133",4,4 | ||
"local-1623281204390","",1,"createOrReplaceTempView at QualificationInfoUtils.scala:133",2,2 | ||
"local-1623281204390","",0,"json at QualificationInfoUtils.scala:130",1209,0 | ||
"local-1623281204390","",6,"json at QualificationInfoUtils.scala:130",110,0 | ||
"local-1623281204390","",3,"json at QualificationInfoUtils.scala:130",108,0 |
Oops, something went wrong.