Incorrect arguments to mysqld_stmt_execute after AWS RDS minor version upgrade (MySQL)

[LajosPolya]

Version

4.3.8

Context

Last week, I ran into an issue where SQL queries started to fail even though there was no recent (in the past year) production deployment. The only non-constant I could find was that our AWS RDS DB cluster (MySQL compatible) had a minor version upgrade nine minutes before the failures began to occur. The AWS RDS cluster upgraded from 8.0.mysql_aurora.3.08.1 to 8.0.mysql_aurora.3.08.2. The new version is compatible with MySQL version 8.0.39.

ERROR 1 --- [ntloop-thread-0] org.hibernate.reactive.errors : HR000057: Failed to execute statement [select member0_.id as id1_5_0_, member0_.active as active2_5_0_, member0_.date_created as date_cre3_5_0_, member0_.last_modified as last_mod4_5_0_, member0_.name as name5_5_0_, member0_.tlx_member_id as tlx_memb6_5_0_ from member member0_ where member0_.id=?]: could not load an entity: [tv.ex.api.entity.Member#1]
java.util.concurrent.CompletionException: io.vertx.mysqlclient.MySQLException: Incorrect arguments to mysqld_stmt_execute
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:936) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[na:na]
at io.vertx.core.Future.lambda$toCompletionStage$3(Future.java:386) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.sqlclient.impl.QueryResultBuilder.tryFail(QueryResultBuilder.java:116) ~[vertx-sql-client-4.3.8.jar!/:4.3.8]
at io.vertx.core.Promise.fail(Promise.java:89) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.Promise.handle(Promise.java:53) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.Promise.handle(Promise.java:29) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:163) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:66) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.PromiseImpl.onFailure(PromiseImpl.java:54) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:43) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.sqlclient.impl.command.CommandResponse.fire(CommandResponse.java:46) ~[vertx-sql-client-4.3.8.jar!/:4.3.8]
at io.vertx.sqlclient.impl.SocketConnectionBase.handleMessage(SocketConnectionBase.java:292) ~[vertx-sql-client-4.3.8.jar!/:4.3.8]
at io.vertx.sqlclient.impl.SocketConnectionBase.lambda$init$0(SocketConnectionBase.java:105) ~[vertx-sql-client-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.impl.ContextBase.emit(ContextBase.java:239) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:390) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[vertx-core-4.3.8.jar!/:4.3.8]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.vertx.mysqlclient.impl.codec.MySQLEncoder.handleCommandResponse(MySQLEncoder.java:63) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.CommandCodec.handleErrorPacketPayload(CommandCodec.java:124) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.ExtendedQueryCommandBaseCodec.handleInitPacket(ExtendedQueryCommandBaseCodec.java:42) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.QueryCommandBaseCodec.decodePayload(QueryCommandBaseCodec.java:56) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.ExtendedQueryCommandCodec.decodePayload(ExtendedQueryCommandCodec.java:108) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.MySQLDecoder.decodePackets(MySQLDecoder.java:72) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.MySQLDecoder.channelRead(MySQLDecoder.java:48) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.87.Final.jar!/:4.1.87.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.87.Final.jar!/:4.1.87.Final]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: io.vertx.mysqlclient.MySQLException: Incorrect arguments to mysqld_stmt_execute
at io.vertx.mysqlclient.impl.codec.CommandCodec.decodeErrorPacketPayload(CommandCodec.java:134) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
at io.vertx.mysqlclient.impl.codec.CommandCodec.handleErrorPacketPayload(CommandCodec.java:123) ~[vertx-mysql-client-4.3.8.jar!/:4.3.8]
... 27 common frames omitted

Additional Info
I'm using Hibernate Reactive Core 1.1.9.Final

What's interesting about the AWS RDS upgrade is that in the logs it states:

Attempt to upgrade the database instance 'xxx-prod-1' with zero downtime finished. The process took 3184 ms, 83 connections preserved, 0 connections dropped. See the database error log for details.

I have a hunch that part of the issue has to do with preserving the open connections but I'm not sure, that's just a guess.

Steps to reproduce

1. Run a query
2. Do an AWS RDS minor version upgrade
3. Run the query again, expect a failure

Do you have a reproducer?

No response

[tsegismont]

@LajosPolya can you reproduce with the latest version of Vert.x 4? That is 4.5.20 at this moment.

[LajosPolya]

I'll write up a test for it next week and get back to you

[LajosPolya]

It's taking me longer than expected to bring up a small working exmaple to reproduce the issue, but I'll have something soon

[LajosPolya]

I'm running into a slight issue. AWS RDS isn't allowing me to deploying an older minor version of the database so I have to deploy a more recent version, wait for it to update and see what happens. So it may take a few weeks before I can reproduce the issue

[tsegismont]

No worries