Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ScanExecutorPT exceptions #163

Merged
merged 4 commits into from
Oct 5, 2021
Merged

Conversation

DomGarguilo
Copy link
Member

Fixes #162

  • Replace old hint method with new scan_type method of selecting executors

@DomGarguilo
Copy link
Member Author

I'm not certain that these changes are preserving the exact functionality of the old code. Also, there are a few minor changes that I will include in this PR once all fix-related changes are deemed correct as to not clutter the diff.

@DomGarguilo
Copy link
Member Author

@keith-turner, it looks like you implemented these changes in apache/accumulo#972 so I was wondering if you might have some thoughts regarding these changes.

Copy link
Contributor

@keith-turner keith-turner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DomGarguilo your changes look correct to me. I made some suggestions to simplify the config and use more descriptive names for scan_type.

Copy link
Contributor

@Manno15 Manno15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. The test completes as expected. I did notice a few small corrections to make with other parts of the test but I will wait until you push your other changes up just in case they are already handled.

Comment on lines 98 to 100
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p1", "se1");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p2", "se1");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se2p1", "se2");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we just set the executor property then that will be the default for all scan types, so do not need to set it twice. When scan_type equals dedicated, it will override it to se2.

Suggested change
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p1", "se1");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se1p2", "se1");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.se2p1", "se2");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor", "se1");
props.put(Property.TABLE_SCAN_DISPATCHER_OPTS.getKey() + "executor.dedicated", "se2");

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the test, se1 is used at two different priorities though so I thought we would need to define two scan types: se1 with priority 1 and the other se1 with priority 2. @keith-turner

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you do need two scan types. This per table dispatcher does not care about prio, it just dispatches scans to the executor se1 or se2. The executor se1 has further config that sets the prio based on scan type, which is where the two diff scan types matter. So we have it dispatch everything to executor se1 unless the scan type is dedicated in which case it dispatches to se2.

@DomGarguilo
Copy link
Member Author

Changes look good. The test completes as expected. I did notice a few small corrections to make with other parts of the test but I will wait until you push your other changes up just in case they are already handled.

@Manno15, I included the other changes I had in mind in da3635d. Let me know if you have some more improvements.

@DomGarguilo DomGarguilo merged commit c2c2408 into apache:main Oct 5, 2021
@DomGarguilo DomGarguilo deleted the scanExecutorPT branch October 5, 2021 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ScanExecutorPT fails to run. Throws AccumuloServerException
3 participants