[CI] GetSettingsBlocksIT testGetSettingsWithBlocks failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9521 / sles-15_platform-support-unix
• elasticsearch-periodic-platform-support #9521 / rocky-9_platform-support-unix

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.indices.settings.GetSettingsBlocksIT.testGetSettingsWithBlocks" -Dtests.seed=40123F2D0521DEB9 -Dtests.locale=es-CR -Dtests.timezone=America/Aruba -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
#1:
	in [elasticsearch[main-cluster-0][search_coordination][T#1]][testTimeoutSetting]
	org.elasticsearch.search.SearchHits.deallocate(SearchHits.java:258)
	org.elasticsearch.search.SearchHits.decRef(SearchHits.java:248)
	org.elasticsearch.action.search.SearchResponse.decRef(SearchResponse.java:243)
	org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:469)
	org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:425)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067)
	org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.ja
[truncated]

Issue Reasons:

• [main] 2 failures in test testGetSettingsWithBlocks (1.2% fail rate in 164 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Judging from the stack trace, I do not think this is related to the ingest node. Perhaps more related to the search foundations side since it's coming from SearchHits

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

Just looked at this briefly for triaging:

• elasticsearch-periodic-platform-support #9521 / sles-15_platform-support-unix

The cause of the reported Leak doesn't really seem to be in "GetSettingsBlocksIT testGetSettingsWithBlocks" but as far as I can tell from the Leak message already happens in another test, namely RemoteSearchForceConnectTimeoutIT testTimeoutSetting :

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
#1:
	in [elasticsearch[main-cluster-0][search_coordination][T#1]][testTimeoutSetting]

That test fails because of some missed assertion apparently due to:

2025-07-22T00:39:33,486][WARN ][o.e.d.n.Node             ][testTimeoutSetting] data_stream.dataset="elasticsearch.deprecation" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="shared-data-path" message="setting [path.shared_data] is deprecated and will be removed in a future release"
[2025-07-22T00:39:33,488][WARN ][o.e.t.ProxyConnectionStrategy][main-cluster-0][generic][T#2] failed to open any proxy connections to cluster [cluster-a]
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:20143] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1168) ~[main/:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415) ~[main/:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327) ~[main/:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:352) ~[main/:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:264) ~[main/:?]
	at org.elasticsearch.transport.netty4.Netty4Utils.lambda$addListener$2(Netty4Utils.java:237) ~[transport-netty4-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:326) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:342) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.118.Final.jar:4.1.118.Final]
	at java.lang.Thread.run(Thread.java:1447) ~[?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:20143
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:628) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:1046) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[netty-transport-4.1.118.Final.jar:4.1.118.Final]
	... 7 more

I see we already have an issue for this.

[cbuescher]

• elasticsearch-periodic-platform-support #9521 / rocky-9_platform-support-unix

There are three failures here in parallel it seems, one of which is similar to the error above from RemoteSearchForceConnectTimeoutIT testTimeoutSetting. The Leaks reported later in e.g. CircuitBreakerNoopIT and GetSettingsBlockIT are caused by that I think

[cbuescher]

In terms of failure history, the suite has only failed the two times reported here in the last 2 months. Since "RemoteSearchForceConnectTimeoutIT testTimeoutSetting" which seems to be the cause has been muted on Jul 23 I don't expect this to happen again. Closing for that reason, we track the underlying problem in #131656