[BUG] all shards failed after node upgrade to 2.11 #11491

Jakob3xD · 2023-12-06T14:06:12Z

Describe the bug
Starting with the upgrade to 2.11 and 2.11.1 our clusters have issues recovering shards.
There were two cases. One case were the cluster it self did not get green and some shards failed on one node. Even clearing the Data of the Node did not help. Only a full cluster reboot.
In the second case the cluster it self gets green but the other cluster making search request via remote search on the upgraded cluster spams errors with failing shards.

Only doing a full cluster reboot of the upgraded cluster helps to solve this issue.

Following is a trace of the main cluster running OpenSearch 2.10 that does the remote search on the upgraded cluster. The cause is always the same -84 is not a valid id. In both cases for all affected indices.

[2023-12-06T13:43:55,865][WARN ][r.suppressed             ] [node1] path: /%2A%3Abmc-info-v1%2A,bmc-info-v1%2A/_search, params: {size=1, index=*:bmc-info-v1*,bmc-info-v1*, sort=@timestamp:desc}
org.opensearch.transport.RemoteTransportException: [error while communicating with remote cluster [cluster3]]
Caused by: org.opensearch.transport.RemoteTransportException: [cluster3-node-3][172.27.1.3:9300][indices:data/read/search]
Caused by: org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:706) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:379) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:503) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:301) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:755) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:903) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:402) [opensearch-security-2.10.0.0.jar:2.10.0.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1526) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:438) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:436) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:428) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:166) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.10.0.jar:2.10.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.97.Final.jar:4.1.97.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.OpenSearchException: java.lang.IllegalArgumentException: -84 is not a valid id
	at org.opensearch.security.support.Base64CustomHelper.deserializeObject(Base64CustomHelper.java:134) ~[?:?]
	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:46) ~[?:?]
	at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:787) ~[?:?]
	at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:208) ~[?:?]
	at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:164) ~[?:?]
	at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:72) ~[?:?]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:185) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:774) ~[?:?]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.IllegalArgumentException: -84 is not a valid id
	at org.opensearch.security.support.Base64CustomHelper$CustomSerializationFormat.fromId(Base64CustomHelper.java:61) ~[?:?]
	at org.opensearch.security.support.Base64CustomHelper.deserializeObject(Base64CustomHelper.java:115) ~[?:?]
	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:46) ~[?:?]
	at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:787) ~[?:?]
	at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:208) ~[?:?]
	at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:164) ~[?:?]
	at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:72) ~[?:?]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:185) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:774) ~[?:?]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

To Reproduce
Had no time yet to test it with clean OpenSearch. Will add it later.
In my current setup it always starts once I upgrade a cluster doing node by node reboots.

Expected behavior
I expect node upgrade to work one by one without failing shards.

Plugins

opensearch-alerting                  2.10.0.0
opensearch-anomaly-detection         2.10.0.0
opensearch-asynchronous-search       2.10.0.0
opensearch-cross-cluster-replication 2.10.0.0
opensearch-custom-codecs             2.10.0.0
opensearch-geospatial                2.10.0.0
opensearch-index-management          2.10.0.0
opensearch-job-scheduler             2.10.0.0
opensearch-knn                       2.10.0.0
opensearch-ml                        2.10.0.0
opensearch-neural-search             2.10.0.0
opensearch-notifications             2.10.0.0
opensearch-notifications-core        2.10.0.0
opensearch-observability             2.10.0.0
opensearch-reports-scheduler         2.10.0.0
opensearch-security                  2.10.0.0
opensearch-security-analytics        2.10.0.0
opensearch-sql                       2.10.0.0
prometheus-exporter                  2.10.0.0
repository-s3                        2.10.0

Host/Environment (please complete the following information):

OS: Ubuntu
Version 22.04

OpenSearch runs within docker using the base OpenSearch image with removed opensearch-performance-analyzer and added s3 plugin.

Additional context

The text was updated successfully, but these errors were encountered:

reta · 2023-12-06T14:08:09Z

@cwperks @scrawfor99 @peternied folks, this seems to be related to security plugin:

	at org.opensearch.security.support.Base64CustomHelper$CustomSerializationFormat.fromId(Base64CustomHelper.java:61) ~[?:?]

cwperks · 2023-12-06T16:22:24Z

@reta There is a fix for this that was merged, but it didn't make it into 2.11.1. opensearch-project/security#3769

reta · 2023-12-06T17:09:24Z

@reta There is a fix for this that was merged, but it didn't make it into 2.11.1. opensearch-project/security#3769

Thanks @cwperks , @peternied could we move it to security repo (from core)? thank you

cwperks · 2023-12-06T18:24:57Z

@reta this can be closed as a duplicate of: opensearch-project/security#3771

spliakos · 2024-10-09T10:34:44Z

Guys, we got the same issue when upgrading from 2.12.0 to 2.17.0 :)
Truth to be said, this happened only on one of our clusters, the rest were OK during the upgrade process

reta · 2024-10-09T12:47:03Z

Thanks @spliakos, please comment on opensearch-project/security#3771, this is not an OpenSearch core issue.

cwperks · 2024-10-09T13:16:44Z

@spliakos This is known to affect rolling upgrades between 2.12,2.13,2.14 to >2.14.

A fix is targeted for 2.18: opensearch-project/security#4741

Jakob3xD added bug Something isn't working untriaged labels Dec 6, 2023

reta closed this as completed Dec 6, 2023

waqarsky mentioned this issue Jan 31, 2024

[BUG] High CPU usage with flat_object field type #12030

Closed

Dhruvan1217 mentioned this issue Feb 28, 2024

Node fails to perform data write on replica shards while upgrading to OS 2.12 opensearch-project/security#4085

Closed

Jakob3xD mentioned this issue Jun 26, 2024

[BUG] Unexpected failure while sending request, -84 is not a valid id opensearch-project/security#4494

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] all shards failed after node upgrade to 2.11 #11491

[BUG] all shards failed after node upgrade to 2.11 #11491

Jakob3xD commented Dec 6, 2023

reta commented Dec 6, 2023

cwperks commented Dec 6, 2023

reta commented Dec 6, 2023

cwperks commented Dec 6, 2023

spliakos commented Oct 9, 2024 •

edited

Loading

reta commented Oct 9, 2024

cwperks commented Oct 9, 2024

[BUG] all shards failed after node upgrade to 2.11 #11491

[BUG] all shards failed after node upgrade to 2.11 #11491

Comments

Jakob3xD commented Dec 6, 2023

Additional context

reta commented Dec 6, 2023

cwperks commented Dec 6, 2023

reta commented Dec 6, 2023

cwperks commented Dec 6, 2023

spliakos commented Oct 9, 2024 • edited Loading

reta commented Oct 9, 2024

cwperks commented Oct 9, 2024

spliakos commented Oct 9, 2024 •

edited

Loading