post the output of dotnet-dump

Here are the biggest allocation that are reported by dumpheap -stat

00007ff9e1bc7a90   192865     14887380 System.String
00007ff9e1bc2990    29332     69153852 System.Int32[]
00007ff9e555c7d0  2224439     71182048 EPiServer.Framework.Dependencies.DependencySorter`1+<>c__DisplayClass2_0[[EPiServer.Framework.Web.Resources.ClientResource, EPiServer.Framework.AspNetCore]]
0000016814a73650     9106    104564936      Free
00007ff9e4f65c88        4    134217904 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
00007ff9e2a85568  8388640    335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings
00007ff9e4f65b00       50   1140461184 EPiServer.Framework.Web.Resources.ClientResource[]

I am trying to examine the content of the memory but it is taking forever.

Took

00007ff9e2a85568  8388640    335545600 EPiServer.Framework.Web.Resources.ClientResourceSettings

>dumpheap -mt 00007ff9e2a85568

0000016815fedcd8 00007ff9e2a85568       40
0000016815fedd00 00007ff9e2a85568       40
0000016815fedd28 00007ff9e2a85568       40
0000016815fedd50 00007ff9e2a85568       40
0000016815fedd78 00007ff9e2a85568       40
....
....

and run gcroot on one of the addresses

>gcroot -all 0000016815fedd78

The below stack is showing up in threads. Seems checking license.

    0000001C403FF160 00007FF9E3977D06 EPiServer.Scheduler.Internal.SchedulerService.Run()
        rbp-10: 0000001c403ff1b0
            ->  0000016915BC2BB0 EPiServer.Scheduler.Internal.WaitableTimer
            ->  0000016915BC2D38 System.Threading.Timer
            ->  0000016915BC2DB0 System.Threading.TimerHolder
            ->  0000016915BC2D50 System.Threading.TimerQueueTimer
            ->  000001681C82EA10 System.Threading.TimerQueueTimer
            ->  0000016A1593B1F8 Microsoft.Extensions.Http.DefaultHttpClientFactory
            ->  0000016A151EC038 Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope
            ->  0000016A15200360 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
            ->  000001681539DDF0 System.Object[]
            ->  00000169150BE400 EPiServer.Framework.Cache.Internal.MemoryPressureMonitor
            ->  00000169150BE3C0 EPiServer.Framework.Cache.Internal.DefaultMemoryPressureEvents
            ->  00000169150E4320 System.EventHandler`1[[EPiServer.Framework.Cache.Internal.MemoryPressureEventArgs, EPiServer.Framework]]
            ->  00000169150E4290 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor
            ->  00000169150E42E0 System.Action`1[[System.Double, System.Private.CoreLib]]
            ->  00000169150E42C8 EPiServer.Framework.Cache.Internal.MemoryCacheCompactor+<>c__DisplayClass8_0
            ->  00000169150E3CB8 Microsoft.Extensions.Caching.Memory.MemoryCache
            ->  00000169150E4068 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016915B80180 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016915B6D968 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]][]
            ->  0000016B15DBE538 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Object, System.Private.CoreLib],[Microsoft.Extensions.Caching.Memory.CacheEntry, Microsoft.Extensions.Caching.Memory]]
            ->  0000016B15DBE3E0 Microsoft.Extensions.Caching.Memory.CacheEntry
            ->  0000016B15DBD170 <Unloaded Type>
            ->  0000016B15D9E7E0 Castle.DynamicProxy.IInterceptor[]
            ->  0000016B15D9E7C0 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptor
            ->  0000016915253A00 EPiServer.DataAbstraction.RuntimeModel.ContentDataInterceptorHandler
            ->  000001691525A090 EPiServer.DataAbstraction.RuntimeModel.Internal.PropertyInterceptorRegistry
            ->  000001691525B2C8 System.EventHandler`1[[System.EventArgs, System.Private.CoreLib]]
            ->  000001691525A4D0 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentDataActivatorSelector
            ->  0000016915259DD8 EPiServer.DataAbstraction.RuntimeModel.Internal.IlGeneratedContentDataActivator
            ->  0000016915256D20 EPiServer.DataAbstraction.RuntimeModel.Internal.ContentModelILCompiler
            ->  0000016915256CB8 EPiServer.DataAbstraction.Internal.BackingTypeResolver
            ->  0000016915256C20 EPiServer.DataAbstraction.Internal.PropertyDefinitionTypeRepository
            ->  00000169150E4938 EPiServer.Events.Internal.RemoteCacheSynchronization
            ->  00000169150E4360 EPiServer.Events.Clients.Internal.EventRegistry
            ->  00000169150E4708 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  00000169150E48F0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  00000169150E47E0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]][]
            ->  00000168150850E8 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Guid, System.Private.CoreLib],[EPiServer.Events.Clients.Event, EPiServer.Events]]
            ->  0000016815084FF8 EPiServer.Events.Clients.Event
            ->  00000169150E46E0 EPiServer.Events.Providers.Internal.ProxyEventBroker
            ->  0000016815080D48 EPiServer.Events.Providers.Internal.EventProviderService
            ->  00000168150803C8 EPiServer.Events.Providers.Internal.EventMessageValidator
            ->  00000168150807D0 EPiServer.Framework.Timers.Internal.SystemTimer
            ->  00000168150807F8 System.Timers.Timer
            ->  0000016815080928 System.Threading.Timer
            ->  00000168150809A0 System.Threading.TimerHolder
            ->  0000016815080940 System.Threading.TimerQueueTimer
            ->  00000169150BE5C0 System.Threading.TimerQueueTimer
            ->  0000016B15145B90 System.Threading.TimerQueueTimer
            ->  000001691514AE30 System.Threading.TimerQueueTimer
            ->  000001681539D340 System.Threading.TimerQueueTimer
            ->  0000016915158180 System.Threading.TimerQueueTimer
            ->  0000016A15920280 System.Threading.TimerQueueTimer
            ->  0000016A15920228 System.Threading.TimerCallback
            ->  0000016A15920150 EPiServer.Licensing.Internal.CloudLicenseMonitor
            ->  0000016A15920200 EPiServer.Licensing.Internal.CloudLicenseValidator
            ->  0000016A16074868 EPiServer.Licensing.Services.LicensedExtendedDataService
            ->  0000016A1511BEA0 EPiServer.Licensing.Services.LicensingServices
            ->  0000016A1511BEB8 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  0000016A1511C0A0 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  0000016A1511BF90 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]][]
            ->  000001681508E050 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
            ->  000001681508E030 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
            ->  000001681508E080 System.Object[]
            ->  0000016815398280 EPiServer.Shell.LicensedExtendedDataService
            ->  0000016A151EBFF0 Microsoft.Extensions.DependencyInjection.ServiceProvider
            ->  0000016A151EC0B8 Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine
            ->  0000016A151EC0D8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder
            ->  0000016A151EC120 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B32018 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B31DB0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]][]
            ->  0000016815B32510 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey, Microsoft.Extensions.DependencyInjection],[Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+GeneratedMethod, Microsoft.Extensions.DependencyInjection]]
            ->  0000016815B1DD90 System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]]
            ->  0000016815B1DBB8 Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder+ILEmitResolverBuilderRuntimeContext
            ->  0000016815B1DBD8 System.Object[]
            ->  0000016B159DD1F0 Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite
            ->  0000016B159DD568 EPiServer.Framework.Web.Resources.Internal.RequiredClientResourceList
            ->  0000016B159DD5C0 System.Collections.Generic.List`1[[EPiServer.Framework.Web.Resources.ClientResourceSettings, EPiServer.Framework.AspNetCore]]
            ->  0000016CAD641020 EPiServer.Framework.Web.Resources.ClientResourceSettings[]
            ->  000001681650D688 EPiServer.Framework.Web.Resources.ClientResourceSettings

Hi Himadri, I strongly suggest you to reach out to Application Support Service. We need to check the memody dump before giving any suggestions. 

Dump analysis shows the client resource allocation issue

https://textdoc.co/MvSkGjPcNXJnDdpa

Himadri, got a shot in the dark answer for you. Have you tried temporarily shutting down the firewall on the server then attempt to start the site?

From the memory dump you provided to the support team, it seems you have a problem with "epi.find.trackingScript" in Footer section. Does it ring a bell?

I searched the whole solution for epi.find.trackingScript, ClientResources.RequireScript, RequireScript, [ClientResourceProvider]. Didn't find anything. I have checked the code using view source in the browser and didn't find anything. I need little more information about what you are seeing in the dump file that indicates that find tracking is used. We haven't implemented that. I can remove find from the application and see if that helps. But, I will still need to know how to fix the issue with find if that is the problem. Thanks for the help.

I checked the ClientResourceSettings and that's everywhere. I suppose you might have some kind of infinitite loop that makes a large number of those objects. Support might need to request your source code for further investigation though. 

I will share the code with support as soon as I get a request through the ticket. I do agree that the application is getting into an infinite loop of some sort allocating client resources. We are not able to pinpoint that. It was not an issue in CMS 11. Things that have changed are that we are using a lot more Async functions like Html.RenderPartialAsync.

Looking deeper, it seems that the clientresourcesettings is registered automatically by Find.UI. you can remove it to test. I will need to consult my colleagues in Find team to see what's wrong

Will remove Find and see what happens. Thanks.

We removed ‘Find’ from the application and the memory leak stopped. We tested the site for the last 24 hours and the issue hasn’t occurred yet. I think ‘Find’ is causing the issue.

Note that you only need to remove Find.UI. But yes if you don't use any Find functionalities, you can remove all Find packages

Hmm, so my memory was rusty. I remember Find.UI to be a separate package but it turned out to be just an assembly in Find.cms. so yes you will need to remove all. Sorry for the confusion. You can however just disable tracking (using episerver:EnableTracking flag) 

I think this is a bug, but to understand where (and how to fix it), we will need to be able to reproduce it 

Yeah, we were not able to remove just Find.UI.