EPiServer.Find.LambdaExpressionExtensions.

EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately

Martin Gustavsson

Vote:

The private static field EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately even when executing the same query and receiving the same result over and over. This will eventually lead to the worker process running out of memory (in our case, quite frequently).

Below is a code snippet that re-creates the behavior.

// The simplest possible query returning the same result every time will still
// increase the number of entries in the cache.

var cacheField = typeof(LambdaExpressionExtensions)
.GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;

cache.Clear();

Debug.Assert(cache.Count == 0);

var findClient = ServiceLocator.Current.GetInstance<IClient>();

// Make the simplest possible query to return one hit.

var firstHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();

Debug.Assert(firstHits.Count == 1);

var firstHitID = firstHits[0].Id;

var firstHitCacheCount = cache.Count;

// The initial query should of course add entries to the cache. About 25 in my case.
Debug.Assert(firstHitCacheCount > 0);

// Make the same query in a loop.
for (var i = 0; i < 10; i++)
{
var loopedHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();

Debug.Assert(loopedHits.Count == 1);

var loopedHitID = loopedHits[0].Id;

Debug.Assert(loopedHitID == firstHitID);

var loopedHitCacheCount = cache.Count;

// Given the same search query and the same result, the cache should not grow.
// It grows with each iteration.
Debug.Assert(loopedHitCacheCount <= firstHitCacheCount);
}

#207645

Sep 27, 2019 12:26

Jacob Rastad

Vote:

Any updates on this from Epi?

We upgraded to 13.2.4 and are experiencing large site stability issues in production after that (issues that didn't show on INTE or PREP).

And for some reason there are breaking changes between 13.0.5 and 13.2.x that prohibits us from downgrading the Nuget package. We even tried to downgrade to 13.2.3 but got the same error message that it wasn't possible due to DB-changes.

What happens is that CPU starts to rise and responsetimes from the instance increase until it's unresponsive and needs to restart.

A large and always growing Dictionary list that needs to be searched each time we trigger a find request is a very likely candidate to be the culprit in this.

#208628

Oct 29, 2019 10:26

Martin Gustavsson

Vote:

The work around we use for the time being is calling the below code after executing a search. Our server CPU usage has since stabilized.

var cacheField = typeof(LambdaExpressionExtensions)
.GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;

if (cache.Count > 500) cache.Clear();

#208629

Oct 29, 2019 10:35

Thom Rosengren

Vote:

We are trying a hotfix where we exchange the cache object. Locally this has brought the cache size down and it doesn't grow when loading the same page. However it needs to be tested thorughly.

From initializer:

var cacheField = typeof(LambdaExpressionExtensions)
                .GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);

            cacheField.SetValue(null, new ConcurrentDictionary<Expression, Delegate>(new CustomExpressionEqualityComparer()));

Comparer:

using EPiServer.Find;
using EPiServer.Find.Helpers;
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Linq.Expressions;

namespace Site.Business.Extensions
{
    public class CustomExpressionEqualityComparer : IEqualityComparer<Expression>
    {
        public bool Equals(Expression expr1, Expression expr2)
        {
            if (expr1 == expr2)
                return true;
            if (expr1.GetType() != expr2.GetType() || expr1.Type != expr2.Type || expr1.NodeType != expr2.NodeType)
                return false;
            LambdaExpression lambdaExpression1 = expr1 as LambdaExpression;
            LambdaExpression lambdaExpression2 = expr2 as LambdaExpression;
            if (lambdaExpression1.IsNotNull() && lambdaExpression2.IsNotNull())
            {
                if (this.ArgumentsEquals<ParameterExpression>(lambdaExpression1.Parameters,
                    lambdaExpression2.Parameters))
                    return this.Equals(lambdaExpression1.Body, lambdaExpression2.Body);
                return false;
            }

            UnaryExpression unaryExpression1 = expr1 as UnaryExpression;
            UnaryExpression unaryExpression2 = expr2 as UnaryExpression;
            if (unaryExpression1.IsNotNull() && unaryExpression2.IsNotNull())
            {
                if (object.Equals((object) unaryExpression1.Method, (object) unaryExpression2.Method))
                    return this.Equals(unaryExpression1.Operand, unaryExpression2.Operand);
                return false;
            }

            BinaryExpression binaryExpression1 = expr1 as BinaryExpression;
            BinaryExpression binaryExpression2 = expr2 as BinaryExpression;
            if (binaryExpression1.IsNotNull() && binaryExpression2.IsNotNull())
            {
                if (object.Equals((object) binaryExpression1.Method, (object) binaryExpression2.Method) &&
                    this.Equals(binaryExpression1.Left, binaryExpression2.Left))
                    return this.Equals(binaryExpression1.Right, binaryExpression2.Right);
                return false;
            }

            MethodCallExpression methodCallExpression1 = expr1 as MethodCallExpression;
            MethodCallExpression methodCallExpression2 = expr2 as MethodCallExpression;
            if (methodCallExpression1.IsNotNull() && methodCallExpression2.IsNotNull())
            {
                if (object.Equals((object) methodCallExpression1.Method, (object) methodCallExpression2.Method) &&
                    this.ArgumentsEquals<Expression>(methodCallExpression1.Arguments, methodCallExpression2.Arguments))
                    return this.Equals(methodCallExpression1.Object, methodCallExpression2.Object);
                return false;
            }

            InvocationExpression invocationExpression1 = expr1 as InvocationExpression;
            InvocationExpression invocationExpression2 = expr2 as InvocationExpression;
            if (invocationExpression1.IsNotNull() && invocationExpression2.IsNotNull())
            {
                if (this.ArgumentsEquals<Expression>(invocationExpression1.Arguments, invocationExpression2.Arguments))
                    return this.Equals(invocationExpression1.Expression, invocationExpression2.Expression);
                return false;
            }

            ConditionalExpression conditionalExpression1 = expr1 as ConditionalExpression;
            ConditionalExpression conditionalExpression2 = expr2 as ConditionalExpression;
            if (conditionalExpression1.IsNotNull() && conditionalExpression2.IsNotNull())
            {
                if (this.Equals(conditionalExpression1.Test, conditionalExpression2.Test) &&
                    this.Equals(conditionalExpression1.IfTrue, conditionalExpression2.IfTrue))
                    return this.Equals(conditionalExpression1.IfFalse, conditionalExpression2.IfFalse);
                return false;
            }

            MemberExpression memberExpression1 = expr1 as MemberExpression;
            MemberExpression memberExpression2 = expr2 as MemberExpression;
            if (memberExpression1.IsNotNull() && memberExpression2.IsNotNull())
            {
                if (object.Equals((object) memberExpression1.Member, (object) memberExpression2.Member))
                    return this.Equals(memberExpression1.Expression, memberExpression2.Expression);
                return false;
            }

            ConstantExpression constantExpression1 = expr1 as ConstantExpression;
            ConstantExpression constantExpression2 = expr2 as ConstantExpression;
            if (constantExpression1.IsNotNull() && constantExpression2.IsNotNull())
                return constantExpression1.Value.Equals(constantExpression2.Value);
            ParameterExpression parameterExpression1 = expr1 as ParameterExpression;
            ParameterExpression parameterExpression2 = expr2 as ParameterExpression;
            if (parameterExpression1.IsNotNull() && parameterExpression2.IsNotNull())
                return string.Equals(parameterExpression1.Name, parameterExpression2.Name, StringComparison.Ordinal);
            NewExpression newExpression1 = expr1 as NewExpression;
            NewExpression newExpression2 = expr2 as NewExpression;
            if (newExpression1.IsNotNull() && newExpression2.IsNotNull() &&
                this.ArgumentsEquals<Expression>(newExpression1.Arguments, newExpression2.Arguments))
                return object.Equals((object) newExpression1.Constructor, (object) newExpression2.Constructor);

            return expr1.ToString() == expr2.ToString();
        }

        public int GetHashCode(Expression obj)
        {
            if (obj.IsNull())
                return -1;
            LambdaExpression lambdaExpression = obj as LambdaExpression;
            if (lambdaExpression.IsNotNull())
                return this.GetArgumentsHashCode<ParameterExpression>(lambdaExpression.Parameters) ^ this.GetHashCode(lambdaExpression.Body);
            UnaryExpression unaryExpression = obj as UnaryExpression;
            if (unaryExpression.IsNotNull())
                return (unaryExpression.Method.IsNotNull() ? unaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(unaryExpression.Operand);
            BinaryExpression binaryExpression = obj as BinaryExpression;
            if (binaryExpression.IsNotNull())
                return (binaryExpression.Method.IsNotNull() ? binaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(binaryExpression.Left) ^ this.GetHashCode(binaryExpression.Right);
            MethodCallExpression methodCallExpression = obj as MethodCallExpression;
            if (methodCallExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(methodCallExpression.Arguments) ^ (methodCallExpression.Method.IsNotNull() ? methodCallExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(methodCallExpression.Object);
            InvocationExpression invocationExpression = obj as InvocationExpression;
            if (invocationExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(invocationExpression.Arguments) ^ this.GetHashCode(invocationExpression.Expression);
            ConditionalExpression conditionalExpression = obj as ConditionalExpression;
            if (conditionalExpression.IsNotNull())
                return this.GetHashCode(conditionalExpression.Test) ^ this.GetHashCode(conditionalExpression.IfFalse) ^ this.GetHashCode(conditionalExpression.IfTrue);
            MemberExpression memberExpression = obj as MemberExpression;
            if (memberExpression.IsNotNull())
                return (memberExpression.Member.IsNotNull() ? memberExpression.Member.GetHashCode() : -1) ^ this.GetHashCode(memberExpression.Expression);
            ConstantExpression constantExpression = obj as ConstantExpression;
            if (constantExpression.IsNotNull())
            {
                if (!constantExpression.Value.IsNotNull())
                    return -1;
                return constantExpression.Value.GetHashCode();
            }
            ParameterExpression parameterExpression = obj as ParameterExpression;
            if (parameterExpression.IsNotNull())
            {
                if (!parameterExpression.Name.IsNotNull())
                    return -1;
                return parameterExpression.Name.GetHashCode();
            }
            NewExpression newExpression = obj as NewExpression;
            if (newExpression.IsNotNull())
                return this.GetArgumentsHashCode<Expression>(newExpression.Arguments) ^ (newExpression.Constructor.IsNotNull() ? newExpression.Constructor.GetHashCode() : -1);
            return obj.ToString().GetHashCode();
        }

        private bool ArgumentsEquals<T>(ReadOnlyCollection<T> args1, ReadOnlyCollection<T> args2) where T : Expression
        {
            if (args1.Count != args2.Count)
                return false;
            for (int index = 0; index < args1.Count; ++index)
            {
                if (!this.Equals((Expression) args1[index], (Expression) args2[index]))
                    return false;
            }

            return true;
        }

        private int GetArgumentsHashCode<T>(ReadOnlyCollection<T> args) where T : Expression
        {
            return args.Aggregate<T, int>(0, (Func<int, T, int>)((hash, next) => hash ^ this.GetHashCode((Expression)next)));
        }
    }
}

#208630

Edited, Oct 29, 2019 10:40

Jacob Rastad

Vote:

We deployed the above code fix yesterday and have seen an insane improvement in CPU-usage and memory usage compared to before.

I've attached a graph of CPU usage and memory usage for our instances.

#208712

Oct 30, 2019 17:07

dada

Vote:

This is fixed in code. Now we're waiting for QA and then release.

Internal reference
FIND-6495 EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately

Bug not available on world yet.

#210480

Edited, Nov 25, 2019 14:41

Madis Vellamäe

Vote:

Any plan when fix for it might be published?

#210572

Edited, Nov 28, 2019 10:36

Damian Pytel

Vote:

We also noticed problem with concurrent dictionary - it takes a lot of memory. Is there any update in that case? Anybody can confirm that workaround provided by Martin works fine? Does it have any negative impact on functionality of epifind?

#210937

Dec 12, 2019 13:24

Madis Vellamäe

Vote:

It worked for us, haven't seen any issues yet.

#210938

Dec 12, 2019 13:34

Damian Pytel

Vote:

I can also confirm that fix proposed by Martin works correct, we have tested it on our production and our memory usage charts started looking as they should. Waiting for official fix from Episerver.

#211253

Edited, Dec 23, 2019 10:05

Quan Mai

Vote:

The bug fix is still under review, so it will be another two weeks at least to be released

#211274

Dec 25, 2019 3:20

Jakub Karcz

Vote:

Do we have any updates about the fix and release planning? Let us know how the review is going and share your expectations about potential release dates.

#216000

Jan 22, 2020 7:34

TJensen

Vote:

We can confirm that we are experiencing this problem in production environment aswell and are interesed in a official bugfix.

#216376

Feb 05, 2020 8:37

Quan Mai

Vote:

There have been difficulties with fixing the bug. If you can provide a memory dump that capture the issue, please send it to me (quan.mai@episerver.com) so we can analyse and work on an effective fix.

#216377

Feb 05, 2020 9:05

TJensen - Feb 05, 2020 9:18

I have sent you an email with more info.

dada

Vote:

This is still being worked on. A new fix is currently under review.

#216922

Feb 11, 2020 8:39

Daniel Forsbakk

Vote:

Any news on this one? I may have encontered this bug. I recently analyzed a memory dump, and there was an unreasonable ammount of objects related to Episerver Find queries (34 million). The Garbage Collector runs more and more often, causing tons of CPU usage and high response times. We have also expierenced instances just going down.

#217734

Feb 28, 2020 14:00

Quan Mai

Vote:

The bug has been fixed (i.e. reviewed and merged), but it will be some time before 13.2.6 gets out of the door (QA and everything)

Would you mind sending me the memory dump? Might be interesting looking into it

#217735

Feb 28, 2020 14:07

Daniel Forsbakk

Vote:

That's great news!

I would be happy to help with a memory dump. The dump is 4.8 GB (~ 400MB compressed). How shall we proceed?

#217736

Edited, Feb 28, 2020 14:11

Quan Mai

Vote:

Perhaps you can zip it and send it via wetransfer ? my email as is above - quan.mai [at] episerver.com. Thanks!

#217738

Feb 28, 2020 14:28

Daniel Forsbakk

Vote:

Sent! Thanks in advance.

#217740

Feb 28, 2020 14:38

Quan Mai - Mar 02, 2020 7:17

I looked into this in it is very clear that the cache is big, so I think the fix will help a lot. We can only wait for Find team, however

dada

Vote:

This is fixed as of Find 13.2.6

https://nuget.episerver.com/package/?id=EPiServer.Find

Changes

https://world.episerver.com/documentation/Release-Notes/?packageFilter=EPiServer.Find&typeFilter=All

#218930

Mar 25, 2020 10:02

Damian Pytel

Vote:

Great news. Thank you for update.

#218931

Mar 25, 2020 10:05