Don't miss out Virtual Happy Hour this Friday (April 26).

Try our conversational search powered by Generative AI!

CMS 12.3 is hanging due to memory leak

Vote:
 

I am dropping this question here to see if anyone has seen this issue. We have migrated a Content Cloud application from CMS 11 to CMS 12.3. The application is consuming all the memory in the server and hanging eventually. This is not happening on the developer's computer.  The difference is in the environment, in this case, the server is running on Windows 2016 and the developers' computers are on Windows 10. ASP.NET Core Runtime version is 5.0.16 in both environments. Unfortunately, I don't have any specific information. I took a memory dump but that didn't reveal a lot of information. I will add more information if I find them

#285236
Aug 09, 2022 12:26
Vote:
 

post the output of dotnet-dump

#285238
Aug 09, 2022 14:24
Vote:
 

Here are the biggest allocation that are reported by dumpheap -stat

00007ff9e1bc7a90   192865     14887380 System.String
00007ff9e1bc2990    29332     69153852 System.Int32[]
00007ff9e555c7d0  2224439     71182048 EPiServer.Framework.Dependencies.DependencySorter`1+<>c__DisplayClass2_0[[EPiServer.Framework.Web.Resources.ClientResource, EPiServer.Framework.AspNetCore]]
0000016814a73650     9106    104564936      Free
00007ff9e4f65c88        4    134217904 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
00007ff9e2a85568  8388640    335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings
00007ff9e4f65b00       50   1140461184 EPiServer.Framework.Web.Resources.ClientResource[]

I am trying to examine the content of the memory but it is taking forever.

#285239
Aug 09, 2022 16:07
Vote:
 

Took

00007ff9e2a85568  8388640    335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings

>dumpheap -mt 00007ff9e2a85568

0000016815fedcd8 00007ff9e2a85568       40
0000016815fedd00 00007ff9e2a85568       40
0000016815fedd28 00007ff9e2a85568       40
0000016815fedd50 00007ff9e2a85568       40
0000016815fedd78 00007ff9e2a85568       40
....
....

and run gcroot on one of the addresses

>gcroot -all 0000016815fedd78

The below stack is showing up in threads. Seems checking license.

    0000001C403FF160 00007FF9E3977D06 EPiServer.Scheduler.Internal.SchedulerService.Run()
        rbp-10: 0000001c403ff1b0
            ->  0000016915BC2BB0 EPiServer.Scheduler.Internal.WaitableTimer
            ->  0000016915BC2D38 System.Threading.Timer
            ->  0000016915BC2DB0 System.Threading.TimerHolder
            ->  0000016915BC2D50 System.Threading.TimerQueueTimer
            ->  000001681C82EA10 System.Threading.TimerQueueTimer
            ->  0000016A1593B1F8 Microsoft.Extensions.Http.DefaultHttpClientFactory
            ->  0000016A151EC038 Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope
            ->  0000016A15200360 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
            ->  000001681539DDF0 System.Object[]
            ->  00000169150BE400 EPiServer.Framework.Cache.Internal.MemoryPressureMonitor
            ->  00000169150BE3C0 EPiServer.Framework.Cache.Internal.DefaultMemoryPressureEvents
            ->  00000169150E4320 System.EventHandler`1[[EPiServer.Framework.Cache.Internal.MemoryPressureEventArgs, EPiServer.Framework]]
            ->  00000169150E4290 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor
            ->  00000169150E42E0 System.Action`1[[System.Double, System.Private.CoreLib]]
            ->  00000169150E42C8 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor+<>c__DisplayClass8_0
            ->  00000169150E3CB8 Microsoft.Extensions.Caching.Memory.MemoryCache
            ->  00000169150E4068 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016915B80180 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016915B6D968 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]][]
            ->  0000016B15DBE538 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016B15DBE3E0 Microsoft.Extensions.Caching.Memory.CacheEntry
            ->  0000016B15DBD170 <Unloaded Type>
            ->  0000016B15D9E7E0 Castle.DynamicProxy.IInterceptor[]
            ->  0000016B15D9E7C0 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptor
            ->  0000016915253A00 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptorHandler
            ->  000001691525A090 EPiServer.DataAbstraction.RuntimeModel.Internal.PropertyInterceptorRegistry
            ->  000001691525B2C8 System.EventHandler`1[[System.EventArgs, System.Private.CoreLib]]
            ->  000001691525A4D0 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentDataActivatorSelector
            ->  0000016915259DD8 EPiServer.DataAbstraction.RuntimeModel.Internal.IlGeneratedContentDataActivator
            ->  0000016915256D20 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentModelILCompiler
            ->  0000016915256CB8 EPiServer.DataAbstraction.Internal.BackingTypeResolver
            ->  0000016915256C20 EPiServer.DataAbstraction.Internal.PropertyDefinitionTypeRepository
            ->  00000169150E4938 EPiServer.Events.Internal.RemoteCacheSynchronization
            ->  00000169150E4360 EPiServer.Events.Clients.Internal.EventRegistry
            ->  00000169150E4708 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  00000169150E48F0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  00000169150E47E0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]][]
            ->  00000168150850E8 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  0000016815084FF8 EPiServer.Events.Clients.Event
            ->  00000169150E46E0 EPiServer.Events.Providers.Internal.ProxyEventBroker
            ->  0000016815080D48 EPiServer.Events.Providers.Internal.EventProviderService
            ->  00000168150803C8 EPiServer.Events.Providers.Internal.EventMessageValidator
            ->  00000168150807D0 EPiServer.Framework.Timers.Internal.SystemTimer
            ->  00000168150807F8 System.Timers.Timer
            ->  0000016815080928 System.Threading.Timer
            ->  00000168150809A0 System.Threading.TimerHolder
            ->  0000016815080940 System.Threading.TimerQueueTimer
            ->  00000169150BE5C0 System.Threading.TimerQueueTimer
            ->  0000016B15145B90 System.Threading.TimerQueueTimer
            ->  000001691514AE30 System.Threading.TimerQueueTimer
            ->  000001681539D340 System.Threading.TimerQueueTimer
            ->  0000016915158180 System.Threading.TimerQueueTimer
            ->  0000016A15920280 System.Threading.TimerQueueTimer
            ->  0000016A15920228 System.Threading.TimerCallback
            ->  0000016A15920150 EPiServer.Licensing.Internal.CloudLicenseMonitor
            ->  0000016A15920200 EPiServer.Licensing.Internal.CloudLicenseValidator
            ->  0000016A16074868 EPiServer.Licensing.Services.LicensedExtendedDataService
            ->  0000016A1511BEA0 EPiServer.Licensing.Services.LicensingServices
            ->  0000016A1511BEB8 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  0000016A1511C0A0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  0000016A1511BF90 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]][]
            ->  000001681508E050 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  000001681508E030 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
            ->  000001681508E080 System.Object[]
            ->  0000016815398280 EPiServer.Shell.LicensedExtendedDataService
            ->  0000016A151EBFF0 Microsoft.Extensions.DependencyInjection.ServiceProvider
            ->  0000016A151EC0B8 Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine
            ->  0000016A151EC0D8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder
            ->  0000016A151EC120 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B32018 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B31DB0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]][]
            ->  0000016815B32510 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B1DD90 System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]]
            ->  0000016815B1DBB8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+ILEmitResolverBuilderRuntimeContext
            ->  0000016815B1DBD8 System.Object[]
            ->  0000016B159DD1F0 Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite
            ->  0000016B159DD568 EPiServer.Framework.Web.Resources.Internal.RequiredClientResourceList
            ->  0000016B159DD5C0 System.Collections.Generic.List`1[[EPiServer.Framework.Web.Resources.ClientResourceSettings, EPiServer.Framework.AspNetCore]]
            ->  0000016CAD641020 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
            ->  000001681650D688 EPiServer.Framework.Web.Resources.ClientResourceSettings

#285245
Aug 09, 2022 19:21
Vote:
 

Hi Himadri, I strongly suggest you to reach out to Application Support Service. We need to check the memody dump before giving any suggestions. 

#285417
Aug 12, 2022 13:21
Himadri - Aug 15, 2022 19:52
Thanks. I am in contact with Optimizely teams.
Quan Mai - Aug 16, 2022 10:46
Is there a ticket number? I can ask my team to look into it, looks like an interesting problem to investigate
Himadri - Aug 16, 2022 14:22
Hi Quan, here is the ticket https://support.optimizely.com/hc/en-us/requests/958472
Please let me know how I can share the memory dump.
Vote:
 

Dump analysis shows the client resource allocation issue

https://textdoc.co/MvSkGjPcNXJnDdpa

#285575
Edited, Aug 15, 2022 20:04
Vote:
 

Himadri, got a shot in the dark answer for you. Have you tried temporarily shutting down the firewall on the server then attempt to start the site?

#285611
Aug 16, 2022 10:05
Vote:
 

From the memory dump you provided to the support team, it seems you have a problem with "epi.find.trackingScript" in Footer section. Does it ring a bell?

#285675
Aug 17, 2022 9:26
Vote:
 

I searched the whole solution for epi.find.trackingScript, ClientResources.RequireScript, RequireScript, [ClientResourceProvider]. Didn't find anything. I have checked the code using view source in the browser and didn't find anything. I need little more information about what you are seeing in the dump file that indicates that find tracking is used. We haven't implemented that. I can remove find from the application and see if that helps. But, I will still need to know how to fix the issue with find if that is the problem. Thanks for the help.

#285676
Aug 17, 2022 11:24
Vote:
 

I checked the ClientResourceSettings and that's everywhere. I suppose you might have some kind of infinitite loop that makes a large number of those objects. Support might need to request your source code for further investigation though. 

#285677
Aug 17, 2022 11:38
Vote:
 

I will share the code with support as soon as I get a request through the ticket. I do agree that the application is getting into an infinite loop of some sort allocating client resources. We are not able to pinpoint that. It was not an issue in CMS 11. Things that have changed are that we are using a lot more Async functions like Html.RenderPartialAsync.

#285682
Aug 17, 2022 12:23
Vote:
 

Looking deeper, it seems that the clientresourcesettings is registered automatically by Find.UI. you can remove it to test. I will need to consult my colleagues in Find team to see what's wrong

#285683
Aug 17, 2022 12:41
Vote:
 

Will remove Find and see what happens. Thanks.

#285684
Aug 17, 2022 12:46
Vote:
 

We removed ‘Find’ from the application and the memory leak stopped. We tested the site for the last 24 hours and the issue hasn’t occurred yet. I think ‘Find’ is causing the issue.

#285740
Aug 18, 2022 11:45
Vote:
 

Note that you only need to remove Find.UI. But yes if you don't use any Find functionalities, you can remove all Find packages

#285749
Aug 18, 2022 15:04
Himadri - Aug 18, 2022 19:12
How can I just remove Find.UI from the project?
Vote:
 

Hmm, so my memory was rusty. I remember Find.UI to be a separate package but it turned out to be just an assembly in Find.cms. so yes you will need to remove all. Sorry for the confusion. You can however just disable tracking (using episerver:EnableTracking flag) 

I think this is a bug, but to understand where (and how to fix it), we will need to be able to reproduce it 

#285755
Aug 18, 2022 21:16
Vote:
 

Yeah, we were not able to remove just Find.UI. 

#285791
Aug 19, 2022 1:38
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.