Here are the biggest allocation that are reported by dumpheap -stat
00007ff9e1bc7a90 192865 14887380 System.String
00007ff9e1bc2990 29332 69153852 System.Int32[]
00007ff9e555c7d0 2224439 71182048 EPiServer.Framework.Dependencies.DependencySorter`1+<>c__DisplayClass2_0[[EPiServer.Framework.Web.Resources.ClientResource, EPiServer.Framework.AspNetCore]]
0000016814a73650 9106 104564936 Free
00007ff9e4f65c88 4 134217904 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
00007ff9e2a85568 8388640 335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings
00007ff9e4f65b00 50 1140461184 EPiServer.Framework.Web.Resources.ClientResource[]
I am trying to examine the content of the memory but it is taking forever.
Took
00007ff9e2a85568 8388640 335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings
>dumpheap -mt 00007ff9e2a85568
0000016815fedcd8 00007ff9e2a85568 40
0000016815fedd00 00007ff9e2a85568 40
0000016815fedd28 00007ff9e2a85568 40
0000016815fedd50 00007ff9e2a85568 40
0000016815fedd78 00007ff9e2a85568 40
....
....
and run gcroot on one of the addresses
>gcroot -all 0000016815fedd78
The below stack is showing up in threads. Seems checking license.
0000001C403FF160 00007FF9E3977D06 EPiServer.Scheduler.Internal.SchedulerService.Run()
rbp-10: 0000001c403ff1b0
-> 0000016915BC2BB0 EPiServer.Scheduler.Internal.WaitableTimer
-> 0000016915BC2D38 System.Threading.Timer
-> 0000016915BC2DB0 System.Threading.TimerHolder
-> 0000016915BC2D50 System.Threading.TimerQueueTimer
-> 000001681C82EA10 System.Threading.TimerQueueTimer
-> 0000016A1593B1F8 Microsoft.Extensions.Http.DefaultHttpClientFactory
-> 0000016A151EC038 Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope
-> 0000016A15200360 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
-> 000001681539DDF0 System.Object[]
-> 00000169150BE400 EPiServer.Framework.Cache.Internal.MemoryPressureMonitor
-> 00000169150BE3C0 EPiServer.Framework.Cache.Internal.DefaultMemoryPressureEvents
-> 00000169150E4320 System.EventHandler`1[[EPiServer.Framework.Cache.Internal.MemoryPressureEventArgs, EPiServer.Framework]]
-> 00000169150E4290 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor
-> 00000169150E42E0 System.Action`1[[System.Double, System.Private.CoreLib]]
-> 00000169150E42C8 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor+<>c__DisplayClass8_0
-> 00000169150E3CB8 Microsoft.Extensions.Caching.Memory.MemoryCache
-> 00000169150E4068 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
-> 0000016915B80180 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
-> 0000016915B6D968 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]][]
-> 0000016B15DBE538 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
-> 0000016B15DBE3E0 Microsoft.Extensions.Caching.Memory.CacheEntry
-> 0000016B15DBD170 <Unloaded Type>
-> 0000016B15D9E7E0 Castle.DynamicProxy.IInterceptor[]
-> 0000016B15D9E7C0 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptor
-> 0000016915253A00 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptorHandler
-> 000001691525A090 EPiServer.DataAbstraction.RuntimeModel.Internal.PropertyInterceptorRegistry
-> 000001691525B2C8 System.EventHandler`1[[System.EventArgs, System.Private.CoreLib]]
-> 000001691525A4D0 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentDataActivatorSelector
-> 0000016915259DD8 EPiServer.DataAbstraction.RuntimeModel.Internal.IlGeneratedContentDataActivator
-> 0000016915256D20 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentModelILCompiler
-> 0000016915256CB8 EPiServer.DataAbstraction.Internal.BackingTypeResolver
-> 0000016915256C20 EPiServer.DataAbstraction.Internal.PropertyDefinitionTypeRepository
-> 00000169150E4938 EPiServer.Events.Internal.RemoteCacheSynchronization
-> 00000169150E4360 EPiServer.Events.Clients.Internal.EventRegistry
-> 00000169150E4708 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
-> 00000169150E48F0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
-> 00000169150E47E0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]][]
-> 00000168150850E8 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
-> 0000016815084FF8 EPiServer.Events.Clients.Event
-> 00000169150E46E0 EPiServer.Events.Providers.Internal.ProxyEventBroker
-> 0000016815080D48 EPiServer.Events.Providers.Internal.EventProviderService
-> 00000168150803C8 EPiServer.Events.Providers.Internal.EventMessageValidator
-> 00000168150807D0 EPiServer.Framework.Timers.Internal.SystemTimer
-> 00000168150807F8 System.Timers.Timer
-> 0000016815080928 System.Threading.Timer
-> 00000168150809A0 System.Threading.TimerHolder
-> 0000016815080940 System.Threading.TimerQueueTimer
-> 00000169150BE5C0 System.Threading.TimerQueueTimer
-> 0000016B15145B90 System.Threading.TimerQueueTimer
-> 000001691514AE30 System.Threading.TimerQueueTimer
-> 000001681539D340 System.Threading.TimerQueueTimer
-> 0000016915158180 System.Threading.TimerQueueTimer
-> 0000016A15920280 System.Threading.TimerQueueTimer
-> 0000016A15920228 System.Threading.TimerCallback
-> 0000016A15920150 EPiServer.Licensing.Internal.CloudLicenseMonitor
-> 0000016A15920200 EPiServer.Licensing.Internal.CloudLicenseValidator
-> 0000016A16074868 EPiServer.Licensing.Services.LicensedExtendedDataService
-> 0000016A1511BEA0 EPiServer.Licensing.Services.LicensingServices
-> 0000016A1511BEB8 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
-> 0000016A1511C0A0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
-> 0000016A1511BF90 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]][]
-> 000001681508E050 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
-> 000001681508E030 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
-> 000001681508E080 System.Object[]
-> 0000016815398280 EPiServer.Shell.LicensedExtendedDataService
-> 0000016A151EBFF0 Microsoft.Extensions.DependencyInjection.ServiceProvider
-> 0000016A151EC0B8 Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine
-> 0000016A151EC0D8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder
-> 0000016A151EC120 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
-> 0000016815B32018 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
-> 0000016815B31DB0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]][]
-> 0000016815B32510 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
-> 0000016815B1DD90 System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]]
-> 0000016815B1DBB8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+ILEmitResolverBuilderRuntimeContext
-> 0000016815B1DBD8 System.Object[]
-> 0000016B159DD1F0 Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite
-> 0000016B159DD568 EPiServer.Framework.Web.Resources.Internal.RequiredClientResourceList
-> 0000016B159DD5C0 System.Collections.Generic.List`1[[EPiServer.Framework.Web.Resources.ClientResourceSettings, EPiServer.Framework.AspNetCore]]
-> 0000016CAD641020 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
-> 000001681650D688 EPiServer.Framework.Web.Resources.ClientResourceSettings
Hi Himadri, I strongly suggest you to reach out to Application Support Service. We need to check the memody dump before giving any suggestions.
Himadri, got a shot in the dark answer for you. Have you tried temporarily shutting down the firewall on the server then attempt to start the site?
From the memory dump you provided to the support team, it seems you have a problem with "epi.find.trackingScript" in Footer section. Does it ring a bell?
I searched the whole solution for epi.find.trackingScript, ClientResources.RequireScript, RequireScript, [ClientResourceProvider]. Didn't find anything. I have checked the code using view source in the browser and didn't find anything. I need little more information about what you are seeing in the dump file that indicates that find tracking is used. We haven't implemented that. I can remove find from the application and see if that helps. But, I will still need to know how to fix the issue with find if that is the problem. Thanks for the help.
I checked the ClientResourceSettings and that's everywhere. I suppose you might have some kind of infinitite loop that makes a large number of those objects. Support might need to request your source code for further investigation though.
I will share the code with support as soon as I get a request through the ticket. I do agree that the application is getting into an infinite loop of some sort allocating client resources. We are not able to pinpoint that. It was not an issue in CMS 11. Things that have changed are that we are using a lot more Async functions like Html.RenderPartialAsync.
Looking deeper, it seems that the clientresourcesettings is registered automatically by Find.UI. you can remove it to test. I will need to consult my colleagues in Find team to see what's wrong
We removed ‘Find’ from the application and the memory leak stopped. We tested the site for the last 24 hours and the issue hasn’t occurred yet. I think ‘Find’ is causing the issue.
Hmm, so my memory was rusty. I remember Find.UI to be a separate package but it turned out to be just an assembly in Find.cms. so yes you will need to remove all. Sorry for the confusion. You can however just disable tracking (using episerver:EnableTracking flag)
I think this is a bug, but to understand where (and how to fix it), we will need to be able to reproduce it
I am dropping this question here to see if anyone has seen this issue. We have migrated a Content Cloud application from CMS 11 to CMS 12.3. The application is consuming all the memory in the server and hanging eventually. This is not happening on the developer's computer. The difference is in the environment, in this case, the server is running on Windows 2016 and the developers' computers are on Windows 10. ASP.NET Core Runtime version is 5.0.16 in both environments. Unfortunately, I don't have any specific information. I took a memory dump but that didn't reveal a lot of information. I will add more information if I find them