[BUG] OpenSearchSecurityException[_opendistro_security_dls_query does not match (SG 900D)] occurs intermittently across nodes when using DLS with role variable substitution `(${user.roles})`.

[Ganesh-RB]

What is the bug?

OpenSearchSecurityException _opendistro_security_dls_query does not match (SG 900D) occurs intermittently across nodes when using Document Level Security (DLS) with role variable substitution (${user.roles}).

Issue Details
As current implementation for User class stores the roles as

    private final Set<String> roles = Collections.synchronizedSet(new HashSet<String>());

HashSet doesn't guarantee anything about the order of the elements ref

It makes no guarantees as to the iteration order of the set; in particular, it does not guarantee that the order will remain constant over time

After AuthN we serialize the user Object and add it in the thread context, when the request get transport to other node we deserialize the user object from thread context, After this serialization and deserialization the order of roes can change in some cases.

Example

• User Object on Node which initiates index setting transport call :

[2026-02-18T16:40:48,850][DEBUG][o.o.s.p.PrivilegesEvaluator] [7e2a1ff976e2ff41ccc8ef7bc9b3123f] ganesh@opensearch.org|admin,read-only,kibana_user||WRITE

• Deserialized User Object on different node, in which the roles order has changed

[2026-02-18T16:40:51,707][DEBUG][o.o.s.p.PrivilegesEvaluator] [4345c938a87205b3bef3303b8e6ee23e] ganesh@opensearch.org|kibana_user,admin,read-only||WRITE

For the usecase where we use a DLS query with dynamic attribute substitution, the computed DLSQuery on both nodes will be different and we have a check in DLSFLSValveImpl (till OS 2.18) to validate if both nodes computed the same DLSQuery otherwise we throw Exception

                if (threadContext.getHeader(ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER) != null) {
                    Object deserializedDlsQueries = Base64Helper.deserializeObject(
                        threadContext.getHeader(ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER),
                        threadContext.getTransient(ConfigConstants.USE_JDK_SERIALIZATION)
                    );
                    if (!dlsQueries.equals(deserializedDlsQueries)) {
                        throw new OpenSearchSecurityException(
                            ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER + " does not match (SG 900D)"
                        );
                    }
                } 

Steps to Reproduce

1. Create a cluster with a dedicated master node and multiple data nodes
2. Configure an internal user with backend_roles as:
   • "admin", "read-only", "kibana_user"
3. Create a role (demo-role) and map it to the above user with:
   • An index pattern (abc*)
   • DLS configuration: "dls": "{\"terms\":{\"organization.name\":[${user.roles}]}}"
4. Execute a index settings request
5. Observe: The error occurs intermittently when requests are routed to different nodes

Expected Behavior

Environment

Component	Version
OpenSearch	2.15

Do you have any additional context?

Stack Trace

RemoteTransportException[[7e2a1ff976e2ff41ccc8ef7bc9b3123f][12.12.12.54:9300][indices:monitor/settings/get]]; nested: OpenSearchSecurityException[_opendistro_security_dls_query does not match (SG 900D)];
Caused by: OpenSearchSecurityException[_opendistro_security_dls_query does not match (SG 900D)]
        at org.opensearch.security.configuration.DlsFlsValveImpl.setDlsHeaders(DlsFlsValveImpl.java:450)
        at org.opensearch.security.configuration.DlsFlsValveImpl.invoke(DlsFlsValveImpl.java:178)
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:390)
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:133)
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:129)

        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114)
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:210)
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:323)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:158)
        at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:852)
        at org.opensearch.ratelimitting.admissioncontrol.transport.AdmissionControlTransportHandler.messageReceived(AdmissionControlTransportHandler.java:68)
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:271)
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:144)
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:127)
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770)
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175)
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150)
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115)
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:840)

[cwperks]

@Ganesh-RB this would not be the case in latest versions of the plugin.

This error comes about when there's a modification to a role definition (or additional indices/aliases) added when a user may have Dls/Fls/FieldMasking restrictions. Its usually seen on long running ops like reindex and scroll.

[cwperks]

Problem is if 2 nodes compute different restrictions you get this error and that can happen during long running ops.

[cwperks]

[Triage] can you put some information from the deep dive about how the order for roles was serialized in a different order on 2 different nodes?

Are you planning to raise a PR (if anything for 2.19 for this?)

[Ganesh-RB]

Updated the ticket description with more details, this issue is present till OS 2.18

from the OS 2.19 the validation check is removed in this PR #5482

I will raise the PR with the fIx, can I raise the PR on mainline and backport it to the 2.x and 3.x?