High memory usage

[tudorFrizeru]

Version Information
AKKA 1.5.16
Modules:
AKKA.Persistence.SQL 1.5.13,
AKKA Streams 1.5.16,
AKKA Serilog 1.5.12.1

Describe the performance issue
After migrating to AKKA 1.5, .NET 8 and the new AKKA.Persistence.SQL, the memory usage has increased to 1 GB (sometimes 2 GB between separate runs, with no changes).

Before the upgrade, the project was running AKKA 1.4.49, .NET 6 and the AKKA.Persistence.SqlServer 1.4.35 and the memory allocation was flattening at 400 MB (mainly due to asp.net core hosting).

I did the upgrade performing the complete database schema upgrade with the migration of Tags to the seaprate table and enabling the UUID in the EventJournal.

Data and Specs

Memory dump, sample persistence configuration and a screenshot with the diagnostic events available here

Expected behavior
Similar memory alocation as before

Actual behavior
Memory allocation has increased to 1 GB, and doesn't increase further (as opposed to 400 MB before the upgrade), and sometimes for unexplained reasons it caps at 2 GB.

Environment
Running on Windows 11, .NET 8, SQL Server 2022, hosting the ActorSystem in an asp.net core environment.
Not using the new AKKA Hosting, for historical reasons I create the ActorSystem through a delegate when the asp.net server starts.
No Cluster, no Remoting.

Additional context
Since the migraiton, I keep seeing this line in the debug console log, every 1-2 seconds:

[22:54:19 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [9]
[22:54:21 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [10]
[22:54:22 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [11]
[22:54:27 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [12]
[22:54:29 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [13]
[22:54:30 DBG] Start timer [Akka.Persistence.Sql.Query.InternalProtocol.QueryOrderingIdsTimerKey] with generation [14]

I don't know if it is relevant, the Serilog also has a sink to a Loki instance, in addition to the Console sink.

[to11mtm]

@tudorFrizeru I've taken a look at your dump file and it appears that the growth may be in/around unmanaged memory which should not typically be an issue with Akka.Persistence.Sql.

If I had to guess, I'm betting this is related to a couple of bugs in 8.0 around Native thread local storage cleanup (dotnet/runtime#95722) that won't get released till the 2024/02/13 (about a week from now). Examples of other people running into similar issues on .NET8: dotnet/runtime#96581
, dotnet/runtime#95922

Will keep digging in repo code however.

[tudorFrizeru]

@to11mtm thank you for your quick response! I downgraded to .net 6 again and the problem remains the same.

Here's a screenshot with the managed memory allocations.
I can upload a new memory dump file if required.

[to11mtm]

@tudorFrizeru If you could that may help, there are a few other possibilities and that may narrow things down.

[tudorFrizeru]

@to11mtm I added 2 new dump files at the same link :

• net6 Instrumentation
• net6 Memory Usage

I left them running for a long time, and they both pertain to the .net6 build

[to11mtm]

🤔

Question, is this running in IIS or via console?

[tudorFrizeru]

The application is hosted as a windows service (using TopShelf for window service hosting, running http.sys as a web server). We do not use IIS.
The dump files you have seen above is the service running in console mode, from visual studio.

[tudorFrizeru]

@to11mtm FYI, I managed to isolate the problem to the Akka.Persistence.Sql package : I reverted back to using the Akka.persistence.SqlServer, with .net 8, and I have no memory issues anymore.

I will be happy to test again once there's a fix available!
Thank you for your support and for your great work!

[to11mtm]

@to11mtm FYI, I managed to isolate the problem to the Akka.Persistence.Sql package : I reverted back to using the Akka.persistence.SqlServer, with .net 8, and I have no memory issues anymore.

I will be happy to test again once there's a fix available! Thank you for your support and for your great work!

Thank you for that feedback!

I do have a PR up that I believe will solve some issues.

Was there anything in your app doing an EventsByTag query with a lot of existing 'in-filter' events (i.e. Thousands to tens of thousands) present after the starting offset? One issue I found, there was a missing limit on EventsByTag query, so cases like what I mentioned would potentially cause a lot of pressure on the Large Object Heap (Which would explain, likely quite a bit!)

[Aaronontheweb]

@to11mtm has worked on some memory optimization improvements here around closures: #347

[MagusDraconis]

Hello everyone,
I encountered a similar issue and managed to resolve it in my scenario.
The problem stemmed from loading Entity Framework 8's DbContext in the constructor.

To avoid this, refrain from utilizing the DbContext property directly within the class. Instead, opt to use it within methods.

The recommended approach is to instantiate a new DbContext within each method call. You can achieve this either by creating a new instance explicitly (new DbContext()), or by retrieving it from the services (services.GetService()).

This practice ensures that each method operates with a fresh DbContext, preventing potential conflicts and ensuring better management of resources.

[Aaronontheweb]

The problem stemmed from loading Entity Framework 8's DbContext in the constructor.

What does that have to do with this project? We use Linq2Db, not EF.

[MagusDraconis]

The problem stemmed from loading Entity Framework 8's DbContext in the constructor.

What does that have to do with this project? We use Linq2Db, not EF.

I think it has nothing to do with the Entity Framework, but with Net8.0 and the GC. In my case, the dbContext cache was not released by the GC as long as the class was used. In net4.8 it worked as expected. I thought there might be a similar problem with Linq2Db.