AI OnAI Off
Tweaking memory cache to avoid block by gc_heap::try_allocate_more_space
Assuming you are already on latest version (we did have some fix regarding clean up cache entries, like 1 year ago?), we don't have any further optimization on settings regarding cache/memory. Speaking from experience however it seems the underlying problem is somewhere else - something is allocating memory faster than it should. You can always reach out to developer support and ask to escalate. I will try to see if I can squeeze time for a quick look.
Apr 12, 2024 10:05
Johan Kronberg
- Apr 12, 2024 10:55
It's running 12.28. Here is one ticket number: 1385377
From support answers we do seem to encounter this at traffic spikes more than in other situations...
Solution uses IObjectInstanceCache.Insert() a lot from custom code. Noticed after posting that it don't support setting item priority or any of the underlying item options. I guess we could re-configure memory limit at startup.
From support answers we do seem to encounter this at traffic spikes more than in other situations...
Solution uses IObjectInstanceCache.Insert() a lot from custom code. Noticed after posting that it don't support setting item priority or any of the underlying item options. I guess we could re-configure memory limit at startup.
There are some options that you can use to configure default memory cache instance of IObjectInstanceCache. Here is reference link https://world.optimizely.com/documentation/developer-guides/archive/-net-core-preview/CMS/caching/monitoring-memorycache/
You can consider which value for MaxMemoryPressurePercentage and DefaultPollInterval is reasonalbe with your case first.
And here is priority of items in cache will be removed when exceeding threshold
- All expired items.
- Items by priority. Lowest priority items are removed first.
- Least recently used objects.
- Items with the earliest absolute expiration.
- Items with the earliest sliding expiration.
You also can add more your own logic with cleaning up memory by injecting this interface IMemoryPressureEvents
One more thing that we can consider if your site is deployed in DXP and you do not see anything else could be optimized in your code is upgrading your DXP plan for autoscalling
Edited,
Apr 14, 2024 9:44
Johan Kronberg
- Apr 15, 2024 11:06
Thanks! Trying that now, lowering to "DefaultPollInterval": "0:0:20" and "MaxMemoryPressurePrecentage": 70.
Are you sure it is only MemoryCache allocating lots of memory?
If it mostly happens under high traffic, then it could also be caused by inefficient code. Like repeated string concatenations, LINQ to list enumerations, byte array instead of stream, and so on.
These could provoke GC freeing, even when the cache entries are reasonable.
Apr 17, 2024 13:55
I took a very quick look at the memory dumps provided in the ticket and my gut feeling is as if you are using some unmanaged library, most likely some server rendering like Chakra. is that a case? if yes there have been cases when those are source of memory leaks ... A long, long shot of course
Apr 18, 2024 8:41
Johan Kronberg
- Apr 18, 2024 9:04
There is nothing like that... There are a lot of API calls outside of Azure (sometimes slow and with large responses) that go into JSON deserialization, DB and then into memory cache. That's where the comment from support is leading us currently. We have done _lots_ of optimization to other parts.
Personally, I don't think adding/freeing memory is the main fix. Symptoms I see looks more aligned with app just waiting for available thread or crashing due to too many threads used. So fix would probably be getting "stronger" containers with more cores/CPU-power available and/or working through code and remote API:s getting faster responses. Or just adding more instances to raise the ceiling by balancing load...
My feeling is also that modern .NET on Linux is not as stable as on Windows. When we have the choice we for now use Windows on Azure for other apps. Some findings that could support this theory and a GH issue where others are also chosing Windows and getting better results:
Apr 29, 2024 14:42
Anyone have some experience of MemoryCache optimization (priorities or other entry options, compacting manually, set memory limit?) in new dotnet versions (8 on DXP in our case)?
We have support reporting this finding in a memory dump:
- From the dump file, we see the SQL connection has to wait GC to finish (gc_heap::try_allocate_more_space).
Then we got a list of item types in the cache. Sizes reported are reasonable to me but it seems like when app is under stress freeing up memory takes time and blocks other work.