EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately

Vote:
 

The private static field EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately even when executing the same query and receiving the same result over and over. This will eventually lead to the worker process running out of memory (in our case, quite frequently).

Below is a code snippet that re-creates the behavior.

// The simplest possible query returning the same result every time will still
// increase the number of entries in the cache.

var cacheField = typeof(LambdaExpressionExtensions)
    .GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;

cache.Clear();

Debug.Assert(cache.Count == 0);

var findClient = ServiceLocator.Current.GetInstance<IClient>();

// Make the simplest possible query to return one hit.

var firstHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();

Debug.Assert(firstHits.Count == 1);

var firstHitID = firstHits[0].Id;

var firstHitCacheCount = cache.Count;

// The initial query should of course add entries to the cache. About 25 in my case.
Debug.Assert(firstHitCacheCount > 0);

// Make the same query in a loop.
for (var i = 0; i < 10; i++)
{
    var loopedHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();

    Debug.Assert(loopedHits.Count == 1);

    var loopedHitID = loopedHits[0].Id;

    Debug.Assert(loopedHitID == firstHitID);

    var loopedHitCacheCount = cache.Count;

    // Given the same search query and the same result, the cache should not grow.
    // It grows with each iteration.
    Debug.Assert(loopedHitCacheCount <= firstHitCacheCount);
}

#207645
Sep 27, 2019 12:26
Vote:
 

Any updates on this from Epi?

We upgraded to 13.2.4 and are experiencing large site stability issues in production after that (issues that didn't show on INTE or PREP).

And for some reason there are breaking changes between 13.0.5 and 13.2.x that prohibits us from downgrading the Nuget package. We even tried to downgrade to 13.2.3 but got the same error message that it wasn't possible due to DB-changes.

What happens is that CPU starts to rise and responsetimes from the instance increase until it's unresponsive and needs to restart.

A large and always growing Dictionary list that needs to be searched each time we trigger a find request is a very likely candidate to be the culprit in this.

#208628
Oct 29, 2019 10:26
Vote:
 

The work around we use for the time being is calling the below code after executing a search. Our server CPU usage has since stabilized.

var cacheField = typeof(LambdaExpressionExtensions)
    .GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;

if (cache.Count > 500)
    cache.Clear();

#208629
Oct 29, 2019 10:35
Vote:
 

We are trying a hotfix where we exchange the cache object. Locally this has brought the cache size down and it doesn't grow when loading the same page. However it needs to be tested thorughly.

From initializer:

var cacheField = typeof(LambdaExpressionExtensions)
                .GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

            cacheField.SetValue(null, new ConcurrentDictionary<Expression, Delegate>(new CustomExpressionEqualityComparer()));

Comparer:

using EPiServer.Find;
using EPiServer.Find.Helpers;
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Linq.Expressions;

namespace Site.Business.Extensions
{
    public class CustomExpressionEqualityComparer : IEqualityComparer<Expression>
    {
        public bool Equals(Expression expr1, Expression expr2)
        {
            if (expr1 == expr2)
                return true;
            if (expr1.GetType() != expr2.GetType() || expr1.Type != expr2.Type || expr1.NodeType != expr2.NodeType)
                return false;
            LambdaExpression lambdaExpression1 = expr1 as LambdaExpression;
            LambdaExpression lambdaExpression2 = expr2 as LambdaExpression;
            if (lambdaExpression1.IsNotNull() && lambdaExpression2.IsNotNull())
            {
                if (this.ArgumentsEquals<ParameterExpression>(lambdaExpression1.Parameters,
                    lambdaExpression2.Parameters))
                    return this.Equals(lambdaExpression1.Body, lambdaExpression2.Body);
                return false;
            }

            UnaryExpression unaryExpression1 = expr1 as UnaryExpression;
            UnaryExpression unaryExpression2 = expr2 as UnaryExpression;
            if (unaryExpression1.IsNotNull() && unaryExpression2.IsNotNull())
            {
                if (object.Equals((object) unaryExpression1.Method, (object) unaryExpression2.Method))
                    return this.Equals(unaryExpression1.Operand, unaryExpression2.Operand);
                return false;
            }

            BinaryExpression binaryExpression1 = expr1 as BinaryExpression;
            BinaryExpression binaryExpression2 = expr2 as BinaryExpression;
            if (binaryExpression1.IsNotNull() && binaryExpression2.IsNotNull())
            {
                if (object.Equals((object) binaryExpression1.Method, (object) binaryExpression2.Method) &&
                    this.Equals(binaryExpression1.Left, binaryExpression2.Left))
                    return this.Equals(binaryExpression1.Right, binaryExpression2.Right);
                return false;
            }

            MethodCallExpression methodCallExpression1 = expr1 as MethodCallExpression;
            MethodCallExpression methodCallExpression2 = expr2 as MethodCallExpression;
            if (methodCallExpression1.IsNotNull() && methodCallExpression2.IsNotNull())
            {
                if (object.Equals((object) methodCallExpression1.Method, (object) methodCallExpression2.Method) &&
                    this.ArgumentsEquals<Expression>(methodCallExpression1.Arguments, methodCallExpression2.Arguments))
                    return this.Equals(methodCallExpression1.Object, methodCallExpression2.Object);
                return false;
            }

            InvocationExpression invocationExpression1 = expr1 as InvocationExpression;
            InvocationExpression invocationExpression2 = expr2 as InvocationExpression;
            if (invocationExpression1.IsNotNull() && invocationExpression2.IsNotNull())
            {
                if (this.ArgumentsEquals<Expression>(invocationExpression1.Arguments, invocationExpression2.Arguments))
                    return this.Equals(invocationExpression1.Expression, invocationExpression2.Expression);
                return false;
            }

            ConditionalExpression conditionalExpression1 = expr1 as ConditionalExpression;
            ConditionalExpression conditionalExpression2 = expr2 as ConditionalExpression;
            if (conditionalExpression1.IsNotNull() && conditionalExpression2.IsNotNull())
            {
                if (this.Equals(conditionalExpression1.Test, conditionalExpression2.Test) &&
                    this.Equals(conditionalExpression1.IfTrue, conditionalExpression2.IfTrue))
                    return this.Equals(conditionalExpression1.IfFalse, conditionalExpression2.IfFalse);
                return false;
            }

            MemberExpression memberExpression1 = expr1 as MemberExpression;
            MemberExpression memberExpression2 = expr2 as MemberExpression;
            if (memberExpression1.IsNotNull() && memberExpression2.IsNotNull())
            {
                if (object.Equals((object) memberExpression1.Member, (object) memberExpression2.Member))
                    return this.Equals(memberExpression1.Expression, memberExpression2.Expression);
                return false;
            }

            ConstantExpression constantExpression1 = expr1 as ConstantExpression;
            ConstantExpression constantExpression2 = expr2 as ConstantExpression;
            if (constantExpression1.IsNotNull() && constantExpression2.IsNotNull())
                return constantExpression1.Value.Equals(constantExpression2.Value);
            ParameterExpression parameterExpression1 = expr1 as ParameterExpression;
            ParameterExpression parameterExpression2 = expr2 as ParameterExpression;
            if (parameterExpression1.IsNotNull() && parameterExpression2.IsNotNull())
                return string.Equals(parameterExpression1.Name, parameterExpression2.Name, StringComparison.Ordinal);
            NewExpression newExpression1 = expr1 as NewExpression;
            NewExpression newExpression2 = expr2 as NewExpression;
            if (newExpression1.IsNotNull() && newExpression2.IsNotNull() &&
                this.ArgumentsEquals<Expression>(newExpression1.Arguments, newExpression2.Arguments))
                return object.Equals((object) newExpression1.Constructor, (object) newExpression2.Constructor);

            return expr1.ToString() == expr2.ToString();
        }

        public int GetHashCode(Expression obj)
        {
            if (obj.IsNull())
                return -1;
            LambdaExpression lambdaExpression = obj as LambdaExpression;
            if (lambdaExpression.IsNotNull())
                return this.GetArgumentsHashCode<ParameterExpression>(lambdaExpression.Parameters) ^ this.GetHashCode(lambdaExpression.Body);
            UnaryExpression unaryExpression = obj as UnaryExpression;
            if (unaryExpression.IsNotNull())
                return (unaryExpression.Method.IsNotNull() ? unaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(unaryExpression.Operand);
            BinaryExpression binaryExpression = obj as BinaryExpression;
            if (binaryExpression.IsNotNull())
                return (binaryExpression.Method.IsNotNull() ? binaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(binaryExpression.Left) ^ this.GetHashCode(binaryExpression.Right);
            MethodCallExpression methodCallExpression = obj as MethodCallExpression;
            if (methodCallExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(methodCallExpression.Arguments) ^ (methodCallExpression.Method.IsNotNull() ? methodCallExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(methodCallExpression.Object);
            InvocationExpression invocationExpression = obj as InvocationExpression;
            if (invocationExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(invocationExpression.Arguments) ^ this.GetHashCode(invocationExpression.Expression);
            ConditionalExpression conditionalExpression = obj as ConditionalExpression;
            if (conditionalExpression.IsNotNull())
                return this.GetHashCode(conditionalExpression.Test) ^ this.GetHashCode(conditionalExpression.IfFalse) ^ this.GetHashCode(conditionalExpression.IfTrue);
            MemberExpression memberExpression = obj as MemberExpression;
            if (memberExpression.IsNotNull())
                return (memberExpression.Member.IsNotNull() ? memberExpression.Member.GetHashCode() : -1) ^ this.GetHashCode(memberExpression.Expression);
            ConstantExpression constantExpression = obj as ConstantExpression;
            if (constantExpression.IsNotNull())
            {
                if (!constantExpression.Value.IsNotNull())
                    return -1;
                return constantExpression.Value.GetHashCode();
            }
            ParameterExpression parameterExpression = obj as ParameterExpression;
            if (parameterExpression.IsNotNull())
            {
                if (!parameterExpression.Name.IsNotNull())
                    return -1;
                return parameterExpression.Name.GetHashCode();
            }
            NewExpression newExpression = obj as NewExpression;
            if (newExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(newExpression.Arguments) ^ (newExpression.Constructor.IsNotNull() ? newExpression.Constructor.GetHashCode() : -1);
            return obj.ToString().GetHashCode();
        }

        private bool ArgumentsEquals<T>(ReadOnlyCollection<T> args1, ReadOnlyCollection<T> args2) where T : Expression
        {
            if (args1.Count != args2.Count)
                return false;
            for (int index = 0; index < args1.Count; ++index)
            {
                if (!this.Equals((Expression) args1[index], (Expression) args2[index]))
                    return false;
            }

            return true;
        }

        private int GetArgumentsHashCode<T>(ReadOnlyCollection<T> args) where T : Expression
        {
            return args.Aggregate<T, int>(0, (Func<int, T, int>)((hash, next) => hash ^ this.GetHashCode((Expression)next)));
        }
    }
}
#208630
Edited, Oct 29, 2019 10:40
Vote:
 

We deployed the above code fix yesterday and have seen an insane improvement in CPU-usage and memory usage compared to before. 

I've attached a graph of CPU usage and memory usage for our instances.

#208712
Oct 30, 2019 17:07
Vote:
 

This is fixed in code. Now we're waiting for QA and then release.

Internal reference
FIND-6495 EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately

Bug not available on world yet.

#210480
Edited, Nov 25, 2019 14:41
Vote:
 

Any plan when fix for it might be published?

#210572
Edited, Nov 28, 2019 10:36
Vote:
 

We also noticed problem with concurrent dictionary - it takes a lot of memory. Is there any update in that case? Anybody can confirm that workaround provided by Martin works fine? Does it have any negative impact on functionality of epifind?

#210937
Dec 12, 2019 13:24
Vote:
 

It worked for us, haven't seen any issues yet.

#210938
Dec 12, 2019 13:34
Vote:
 

I can also confirm that fix proposed by Martin works correct, we have tested it on our production and our memory usage charts started looking as they should. Waiting for official fix from Episerver.

#211253
Edited, Dec 23, 2019 10:05
Vote:
 

The bug fix is still under review, so it will be another two weeks at least to be released

#211274
Dec 25, 2019 3:20
Vote:
 

Do we have any updates about the fix and release planning? Let us know how the review is going and share your expectations about potential release dates.

#216000
Jan 22, 2020 7:34
Vote:
 

We can confirm that we are experiencing this problem in production environment aswell and are interesed in a official bugfix.

#216376
Feb 05, 2020 8:37
Vote:
 

There have been difficulties with fixing the bug. If you can provide a memory dump that capture the issue, please send it to me (quan.mai@episerver.com) so we can analyse and work on an effective fix. 

#216377
Feb 05, 2020 9:05
TJensen - Feb 05, 2020 9:18
I have sent you an email with more info.
Vote:
 

This is still being worked on. A new fix is currently under review.

#216922
Feb 11, 2020 8:39
Vote:
 

Any news on this one? I may have encontered this bug. I recently analyzed a memory dump, and there was an unreasonable ammount of objects related to Episerver Find queries (34 million). The Garbage Collector runs more and more often, causing tons of CPU usage and high response times. We have also expierenced instances just going down.  

#217734
Feb 28, 2020 14:00
Vote:
 

The bug has been fixed (i.e. reviewed and merged), but it will be some time before 13.2.6 gets out of the door (QA and everything)

Would you mind sending me the memory dump? Might be interesting looking into it 

#217735
Feb 28, 2020 14:07
Vote:
 

That's great news!

I would be happy to help with a memory dump. The dump is 4.8 GB (~ 400MB compressed). How shall we proceed?

#217736
Edited, Feb 28, 2020 14:11
Vote:
 

Perhaps you can zip it and send it via wetransfer ? my email as is above - quan.mai [at] episerver.com. Thanks!

#217738
Feb 28, 2020 14:28
Vote:
 

Sent! Thanks in advance. 

#217740
Feb 28, 2020 14:38
Quan Mai - Mar 02, 2020 7:17
I looked into this in it is very clear that the cache is big, so I think the fix will help a lot. We can only wait for Find team, however
Vote:
 

Great news. Thank you for update. 

#218931
Mar 25, 2020 10:05
This topic was created over six months ago and has been resolved. If you have a similar question, please create a new topic and refer to this one.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.