-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ScanExecutorPT exceptions #163
Conversation
I'm not certain that these changes are preserving the exact functionality of the old code. Also, there are a few minor changes that I will include in this PR once all fix-related changes are deemed correct as to not clutter the diff. |
@keith-turner, it looks like you implemented these changes in apache/accumulo#972 so I was wondering if you might have some thoughts regarding these changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@DomGarguilo your changes look correct to me. I made some suggestions to simplify the config and use more descriptive names for scan_type.
src/main/java/org/apache/accumulo/testing/performance/tests/ScanExecutorPT.java
Outdated
Show resolved
Hide resolved
src/main/java/org/apache/accumulo/testing/performance/tests/ScanExecutorPT.java
Outdated
Show resolved
Hide resolved
src/main/java/org/apache/accumulo/testing/performance/tests/ScanExecutorPT.java
Outdated
Show resolved
Hide resolved
src/main/java/org/apache/accumulo/testing/performance/tests/ScanExecutorPT.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good. The test completes as expected. I did notice a few small corrections to make with other parts of the test but I will wait until you push your other changes up just in case they are already handled.
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p1", "se1"); | ||
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p2", "se1"); | ||
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se2p1", "se2"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we just set the executor property then that will be the default for all scan types, so do not need to set it twice. When scan_type equals dedicated, it will override it to se2.
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p1", "se1"); | |
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p2", "se1"); | |
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se2p1", "se2"); | |
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor", "se1"); | |
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.dedicated", "se2"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the test, se1 is used at two different priorities though so I thought we would need to define two scan types: se1 with priority 1 and the other se1 with priority 2. @keith-turner
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah you do need two scan types. This per table dispatcher does not care about prio, it just dispatches scans to the executor se1 or se2. The executor se1 has further config that sets the prio based on scan type, which is where the two diff scan types matter. So we have it dispatch everything to executor se1 unless the scan type is dedicated in which case it dispatches to se2.
Co-authored-by: Keith Turner <kturner@apache.org>
@Manno15, I included the other changes I had in mind in da3635d. Let me know if you have some more improvements. |
Fixes #162