Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ScanExecutorPT fails to run. Throws AccumuloServerException #162

Closed
DomGarguilo opened this issue Sep 29, 2021 · 6 comments · Fixed by #163
Closed

ScanExecutorPT fails to run. Throws AccumuloServerException #162

DomGarguilo opened this issue Sep 29, 2021 · 6 comments · Fixed by #163
Assignees
@DomGarguilo

Comments

@DomGarguilo
Copy link
Member

ScanExecutorPT fails to complete with the following output:

Shell output 
Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server thor:9997
	at org.apache.accumulo.testing.performance.util.TestExecutor.lambda$stream$0(TestExecutor.java:52)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.LongPipeline.collect(LongPipeline.java:491)
	at java.base/java.util.stream.LongPipeline.summaryStatistics(LongPipeline.java:468)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.runShortScans(ScanExecutorPT.java:199)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.runTest(ScanExecutorPT.java:109)
	at org.apache.accumulo.testing.performance.impl.PerfTestRunner.main(PerfTestRunner.java:51)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server thor:9997
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.accumulo.testing.performance.util.TestExecutor.lambda$stream$0(TestExecutor.java:50)
	... 11 more
Caused by: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server thor:9997
	at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:185)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:110)
	at com.google.common.collect.Iterators.size(Iterators.java:163)
	at com.google.common.collect.Iterables.size(Iterables.java:126)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.scan(ScanExecutorPT.java:156)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.lambda$runShortScans$1(ScanExecutorPT.java:196)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server thor:9997
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:324)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:156)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:174)
	... 9 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:249)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:221)
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:453)
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:317)
	... 11 more
Server logs 
java.lang.IllegalArgumentException: Invalid option heed_hints at 
org.apache.accumulo.core.spi.scan.SimpleScanDispatcher.lambda$init$0(SimpleScanDispatcher.java:105) at 
com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:185) at 
org.apache.accumulo.core.spi.scan.SimpleScanDispatcher.init(SimpleScanDispatcher.java:81) at 
org.apache.accumulo.server.conf.TableConfiguration.createScanDispatcher(TableConfiguration.java:215) at 
org.apache.accumulo.server.conf.TableConfiguration.lambda$new$1(TableConfiguration.java:82) at 
org.apache.accumulo.core.conf.AccumuloConfiguration$DeriverImpl.derive(AccumuloConfiguration.java:482) at 
org.apache.accumulo.server.conf.TableConfiguration.getScanDispatcher(TableConfiguration.java:270) at 
org.apache.accumulo.tserver.ThriftClientHandler.getScanDispatcher(ThriftClientHandler.java:272) at 
org.apache.accumulo.tserver.ThriftClientHandler.continueScan(ThriftClientHandler.java:378) at 
org.apache.accumulo.tserver.ThriftClientHandler.startScan(ThriftClientHandler.java:342) at 
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at 
java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$6(TraceUtil.java:235) at 
com.sun.proxy.$Proxy38.startScan(Unknown Source) at 
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.getResult(TabletClientService.java:2944) at 
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.getResult(TabletClientService.java:2923)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at 
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) at 
org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:114) at
 org.apache.thrift.server.Invocation.run(Invocation.java:18) at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at 
java.base/java.lang.Thread.run(Thread.java:829)
@Manno15
Copy link
Contributor

Manno15 commented Sep 29, 2021

It seems heed_hints was removed as a valid opt for SimpleScanDispatcher in apache/accumulo#972. Seems the PT was not updated after that change took place.

@DomGarguilo
Copy link
Member Author

It seems heed_hints was removed as a valid opt for SimpleScanDispatcher in apache/accumulo#972. Seems the PT was not updated after that change took place.

Thanks, @Manno15! I think I see how these can be replaced now.

@DomGarguilo DomGarguilo self-assigned this Sep 29, 2021
@DomGarguilo DomGarguilo mentioned this issue Sep 30, 2021
Merged
@Manno15
Copy link
Contributor

Manno15 commented Nov 29, 2021

This PT is failing again with the OpenTelemetry changes.

Shell output 
	Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server groot:9997
	at org.apache.accumulo.testing.performance.util.TestExecutor.lambda$stream$0(TestExecutor.java:52)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.LongPipeline.collect(LongPipeline.java:491)
	at java.base/java.util.stream.LongPipeline.summaryStatistics(LongPipeline.java:468)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.runShortScans(ScanExecutorPT.java:200)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.runTest(ScanExecutorPT.java:110)
	at org.apache.accumulo.testing.performance.impl.PerfTestRunner.main(PerfTestRunner.java:51)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server groot:9997
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.accumulo.testing.performance.util.TestExecutor.lambda$stream$0(TestExecutor.java:50)
	... 11 more
Caused by: java.lang.RuntimeException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server groot:9997
	at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:185)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:110)
	at com.google.common.collect.Iterators.size(Iterators.java:163)
	at com.google.common.collect.Iterables.size(Iterables.java:126)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.scan(ScanExecutorPT.java:157)
	at org.apache.accumulo.testing.performance.tests.ScanExecutorPT.lambda$runShortScans$1(ScanExecutorPT.java:197)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server groot:9997
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:332)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:156)
	at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:174)
	... 9 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing startScan
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:249)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:221)
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:479)
	at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:323)
	... 11 more
Server logs 
	java.lang.ClassCastException: class io.opentelemetry.context.Context$$Lambda$243/0x0000000840400040 cannot be cast to class org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer (io.opentelemetry.context.Context$$Lambda$243/0x0000000840400040 and org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer are in unnamed module of loader 'app')
	at org.apache.accumulo.tserver.TabletServerResourceManager.lambda$createPriorityExecutor$1(TabletServerResourceManager.java:179)
	at java.base/java.util.Comparator.lambda$comparing$ea9a8b3a$1(Comparator.java:436)
	at java.base/java.util.concurrent.PriorityBlockingQueue.siftUpUsingComparator(PriorityBlockingQueue.java:377)
	at java.base/java.util.concurrent.PriorityBlockingQueue.offer(PriorityBlockingQueue.java:488)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1347)
	at org.apache.accumulo.core.util.threads.ThreadPools$1.execute(ThreadPools.java:187)
	at org.apache.accumulo.tserver.TabletServerResourceManager.executeReadAhead(TabletServerResourceManager.java:807)
	at org.apache.accumulo.tserver.ThriftClientHandler.continueScan(ThriftClientHandler.java:434)
	at org.apache.accumulo.tserver.ThriftClientHandler.continueScan(ThriftClientHandler.java:366)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$1(TraceUtil.java:204)
	at com.sun.proxy.$Proxy35.continueScan(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$continueScan.getResult(TabletClientService.java:2979)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$continueScan.getResult(TabletClientService.java:2958)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:54)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524)
	at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:116)
	at org.apache.thrift.server.Invocation.run(Invocation.java:18)
	at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207)
	at java.base/java.lang.Thread.run(Thread.java:829)


java.lang.ClassCastException: class io.opentelemetry.context.Context$$Lambda$243/0x0000000840400040 cannot be cast to class org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer (io.opentelemetry.context.Context$$Lambda$243/0x0000000840400040 and org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer are in unnamed module of loader 'app')
	at org.apache.accumulo.tserver.TabletServerResourceManager.lambda$createPriorityExecutor$1(TabletServerResourceManager.java:179)
	at java.base/java.util.Comparator.lambda$comparing$ea9a8b3a$1(Comparator.java:436)
	at java.base/java.util.concurrent.PriorityBlockingQueue.siftUpUsingComparator(PriorityBlockingQueue.java:377)
	at java.base/java.util.concurrent.PriorityBlockingQueue.offer(PriorityBlockingQueue.java:488)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1347)
	at org.apache.accumulo.core.util.threads.ThreadPools$1.execute(ThreadPools.java:187)
	at org.apache.accumulo.tserver.TabletServerResourceManager.executeReadAhead(TabletServerResourceManager.java:807)
	at org.apache.accumulo.tserver.ThriftClientHandler.continueScan(ThriftClientHandler.java:379)
	at org.apache.accumulo.tserver.ThriftClientHandler.startScan(ThriftClientHandler.java:344)
	at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$1(TraceUtil.java:204)
	at com.sun.proxy.$Proxy35.startScan(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.getResult(TabletClientService.java:2944)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.getResult(TabletClientService.java:2923)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:54)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524)
	at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:116)
	at org.apache.thrift.server.Invocation.run(Invocation.java:18)
	at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207)
	at java.base/java.lang.Thread.run(Thread.java:829)

@Manno15 Manno15 reopened this Nov 29, 2021
@Manno15
Copy link
Contributor

Manno15 commented Dec 4, 2021

This issue seems to be somewhat related to issue apache/accumulo#2362. Both cases are due to the priority blocking queue, though this is a slightly different thrown exception.

@Manno15
Copy link
Contributor

Manno15 commented Feb 27, 2022

@DomGarguilo This PT seems to pass on my end now. If you could confirm it passes on your end and then close this issue if so.

@ctubbsii
Copy link
Member

Based on the stack trace, it definitely should have been fixed by apache/accumulo#2442 ; Closing as presumed fixed based on this and previous comments. Please reopen if otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DomGarguilo DomGarguilo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix ScanExecutorPT exceptions DomGarguilo/accumulo-testing
3 participants
@ctubbsii @Manno15 @DomGarguilo