180K products - Random App Pool Recycle

Islam Hamed

Vote:

Hello,

Using EPiServer.Commerce 12.17.2 and EPiServer.CMS 11.11.2:

We have been talking with Microsoft Azure support team for a while now because of a random app pool reycle that takes place on one of our client's website. The client has about 180K products in their online catalog. The plan is to keep adding products up to 1 Million.

Looking into the memory dumps from a high memory prespective, these were the results:

-Bytes in All Heaps      6,804,301,128 <== Managed heap is nearly 7GB
GCHandles                      30,665
GEN 0 Collections              21,086
GEN 1 Collections               7,970
GEN 2 Collections                 454
# Induced GCs                      24 <== GC.Collect() calls

There are millions of EPiServer.* objects flooding the managed heap:
00007fff38fad170   451149     39701112 EPiServer.SpecializedProperties.PropertyXhtmlString
00007fff29dfe840   677449     43356736 EPiServer.Core.PropertyBoolean
00007fff38eecb78  1289409     51576360 EPiServer.Core.ContentReference
00007fff29dfea78   900046     64803312 EPiServer.Core.PropertyFloatNumber
00007fff38fa92e0   677173     70425992 EPiServer.SpecializedProperties.PropertyContentArea
00007fff29c842d0  1823778     72951120 EPiServer.Util.Internal.AutoCompressString
00007fff29c83dc8  1351222     86478208 EPiServer.Core.PropertyDataCollection
00007fff38fc63c0  1125445    108042720 EPiServer.Core.PropertyContentReference
00007fff38f223e8  2025090    162007200 EPiServer.Commerce.SpecializedProperties.PropertyDecimal
00007fff38f248b8  3376284    216082176 EPiServer.Core.PropertyNumber <== 3 million objects
00007fff2a157f78  1351160    217956944 EPiServer.Core.PropertyData[]
00007fff38f20fc0 12381093    990487440 EPiServer.Core.PropertyLongString <== 12 million objects

Any recomenndations on how to locate the modules responsible of this high memory consumption? Whether it is a custom code or an EPi module?

The website has P3V2 App Service Plan.

#217337

Feb 21, 2020 0:16

Quan Mai

Vote:

180k products are not a very big number . It sounds like the site loads too many products than it needs. There is no easy way to spot that, but if you has captured memory dumps before the site restarts then Application support should be able to look into it and provide some guidance

#217338

Feb 21, 2020 2:10

Islam Hamed

Vote:

@Quan Mai: I cannot thank you enough for taking the time to respond to our posts!

The issue was related to our custom products indexing job where we load the whole catalog to custom index it in FIND.

Using your tutorial here of traversing the catalog contributed in resolving the issue: https://leanpub.com/epicommercerecipes/read_sample

This brings another question to my mind related to memory management. I understand there is an expiry time for VariantContent cached in the memory.

But shouldn't the memory forced to free these cached objects if it reaches its maximum capacity? I guess cached objects should have higher priority than any other objects in memory.

Thanks again for the help.

#218304

Mar 10, 2020 19:32

Quan Mai

Vote:

You are very welcome.

Yes in theory asp.net will trim the cache if there is a memory pressure. However you might have setting that keep the threshold high (e.g the cache will not be trimmed until it reaches 95% ) and you might have auto heal rule that restarts the instance if it reaches, says, 90%. Is it the case?

if you are running Cms11+ and are using scheduled job you have a way to reduce cache time

#218312

Mar 10, 2020 20:13

Islam Hamed

Vote:

We turned off auto heal for troubleshooting this issue. The issue was still occuring.

Yes, we do use CMS 11.11.2. My understaning is contents loaded by schedule job have very short expiration time if I am not mistaken

#218313

Mar 10, 2020 20:26

Quan Mai

Vote:

I misread it a little bit, so you are trying to use the recipe to resolve the problem, not that it caused the problem :) . As long as you load content in batches AND you don't have too aggressive memory management thresholds (I would suggest to start with all default settings), it should be fine.

And yes, CMS 11 allows you to reduce the caching time, as if you are indexing the content you just need them once, then discard.

#218314

Mar 10, 2020 21:08

Islam Hamed

Vote:

Wait a second, why don't you have the blue badge below your name?

Do you need help with commerce certifiication?!! :)

#218390

Mar 11, 2020 23:53

Quan Mai - Mar 12, 2020 6:43

Yes definitely! I want that blue and green badges