Any updates on this from Epi?
We upgraded to 13.2.4 and are experiencing large site stability issues in production after that (issues that didn't show on INTE or PREP).
And for some reason there are breaking changes between 13.0.5 and 13.2.x that prohibits us from downgrading the Nuget package. We even tried to downgrade to 13.2.3 but got the same error message that it wasn't possible due to DB-changes.
What happens is that CPU starts to rise and responsetimes from the instance increase until it's unresponsive and needs to restart.
A large and always growing Dictionary list that needs to be searched each time we trigger a find request is a very likely candidate to be the culprit in this.
The work around we use for the time being is calling the below code after executing a search. Our server CPU usage has since stabilized.
var cacheField = typeof(LambdaExpressionExtensions)
.GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);
var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;
if (cache.Count > 500)
cache.Clear();
We are trying a hotfix where we exchange the cache object. Locally this has brought the cache size down and it doesn't grow when loading the same page. However it needs to be tested thorughly.
From initializer:
var cacheField = typeof(LambdaExpressionExtensions)
.GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);
cacheField.SetValue(null, new ConcurrentDictionary<Expression, Delegate>(new CustomExpressionEqualityComparer()));
Comparer:
using EPiServer.Find;
using EPiServer.Find.Helpers;
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Linq.Expressions;
namespace Site.Business.Extensions
{
public class CustomExpressionEqualityComparer : IEqualityComparer<Expression>
{
public bool Equals(Expression expr1, Expression expr2)
{
if (expr1 == expr2)
return true;
if (expr1.GetType() != expr2.GetType() || expr1.Type != expr2.Type || expr1.NodeType != expr2.NodeType)
return false;
LambdaExpression lambdaExpression1 = expr1 as LambdaExpression;
LambdaExpression lambdaExpression2 = expr2 as LambdaExpression;
if (lambdaExpression1.IsNotNull() && lambdaExpression2.IsNotNull())
{
if (this.ArgumentsEquals<ParameterExpression>(lambdaExpression1.Parameters,
lambdaExpression2.Parameters))
return this.Equals(lambdaExpression1.Body, lambdaExpression2.Body);
return false;
}
UnaryExpression unaryExpression1 = expr1 as UnaryExpression;
UnaryExpression unaryExpression2 = expr2 as UnaryExpression;
if (unaryExpression1.IsNotNull() && unaryExpression2.IsNotNull())
{
if (object.Equals((object) unaryExpression1.Method, (object) unaryExpression2.Method))
return this.Equals(unaryExpression1.Operand, unaryExpression2.Operand);
return false;
}
BinaryExpression binaryExpression1 = expr1 as BinaryExpression;
BinaryExpression binaryExpression2 = expr2 as BinaryExpression;
if (binaryExpression1.IsNotNull() && binaryExpression2.IsNotNull())
{
if (object.Equals((object) binaryExpression1.Method, (object) binaryExpression2.Method) &&
this.Equals(binaryExpression1.Left, binaryExpression2.Left))
return this.Equals(binaryExpression1.Right, binaryExpression2.Right);
return false;
}
MethodCallExpression methodCallExpression1 = expr1 as MethodCallExpression;
MethodCallExpression methodCallExpression2 = expr2 as MethodCallExpression;
if (methodCallExpression1.IsNotNull() && methodCallExpression2.IsNotNull())
{
if (object.Equals((object) methodCallExpression1.Method, (object) methodCallExpression2.Method) &&
this.ArgumentsEquals<Expression>(methodCallExpression1.Arguments, methodCallExpression2.Arguments))
return this.Equals(methodCallExpression1.Object, methodCallExpression2.Object);
return false;
}
InvocationExpression invocationExpression1 = expr1 as InvocationExpression;
InvocationExpression invocationExpression2 = expr2 as InvocationExpression;
if (invocationExpression1.IsNotNull() && invocationExpression2.IsNotNull())
{
if (this.ArgumentsEquals<Expression>(invocationExpression1.Arguments, invocationExpression2.Arguments))
return this.Equals(invocationExpression1.Expression, invocationExpression2.Expression);
return false;
}
ConditionalExpression conditionalExpression1 = expr1 as ConditionalExpression;
ConditionalExpression conditionalExpression2 = expr2 as ConditionalExpression;
if (conditionalExpression1.IsNotNull() && conditionalExpression2.IsNotNull())
{
if (this.Equals(conditionalExpression1.Test, conditionalExpression2.Test) &&
this.Equals(conditionalExpression1.IfTrue, conditionalExpression2.IfTrue))
return this.Equals(conditionalExpression1.IfFalse, conditionalExpression2.IfFalse);
return false;
}
MemberExpression memberExpression1 = expr1 as MemberExpression;
MemberExpression memberExpression2 = expr2 as MemberExpression;
if (memberExpression1.IsNotNull() && memberExpression2.IsNotNull())
{
if (object.Equals((object) memberExpression1.Member, (object) memberExpression2.Member))
return this.Equals(memberExpression1.Expression, memberExpression2.Expression);
return false;
}
ConstantExpression constantExpression1 = expr1 as ConstantExpression;
ConstantExpression constantExpression2 = expr2 as ConstantExpression;
if (constantExpression1.IsNotNull() && constantExpression2.IsNotNull())
return constantExpression1.Value.Equals(constantExpression2.Value);
ParameterExpression parameterExpression1 = expr1 as ParameterExpression;
ParameterExpression parameterExpression2 = expr2 as ParameterExpression;
if (parameterExpression1.IsNotNull() && parameterExpression2.IsNotNull())
return string.Equals(parameterExpression1.Name, parameterExpression2.Name, StringComparison.Ordinal);
NewExpression newExpression1 = expr1 as NewExpression;
NewExpression newExpression2 = expr2 as NewExpression;
if (newExpression1.IsNotNull() && newExpression2.IsNotNull() &&
this.ArgumentsEquals<Expression>(newExpression1.Arguments, newExpression2.Arguments))
return object.Equals((object) newExpression1.Constructor, (object) newExpression2.Constructor);
return expr1.ToString() == expr2.ToString();
}
public int GetHashCode(Expression obj)
{
if (obj.IsNull())
return -1;
LambdaExpression lambdaExpression = obj as LambdaExpression;
if (lambdaExpression.IsNotNull())
return this.GetArgumentsHashCode<ParameterExpression>(lambdaExpression.Parameters) ^ this.GetHashCode(lambdaExpression.Body);
UnaryExpression unaryExpression = obj as UnaryExpression;
if (unaryExpression.IsNotNull())
return (unaryExpression.Method.IsNotNull() ? unaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(unaryExpression.Operand);
BinaryExpression binaryExpression = obj as BinaryExpression;
if (binaryExpression.IsNotNull())
return (binaryExpression.Method.IsNotNull() ? binaryExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(binaryExpression.Left) ^ this.GetHashCode(binaryExpression.Right);
MethodCallExpression methodCallExpression = obj as MethodCallExpression;
if (methodCallExpression.IsNotNull())
return this.GetArgumentsHashCode<Expression>(methodCallExpression.Arguments) ^ (methodCallExpression.Method.IsNotNull() ? methodCallExpression.Method.GetHashCode() : -1) ^ this.GetHashCode(methodCallExpression.Object);
InvocationExpression invocationExpression = obj as InvocationExpression;
if (invocationExpression.IsNotNull())
return this.GetArgumentsHashCode<Expression>(invocationExpression.Arguments) ^ this.GetHashCode(invocationExpression.Expression);
ConditionalExpression conditionalExpression = obj as ConditionalExpression;
if (conditionalExpression.IsNotNull())
return this.GetHashCode(conditionalExpression.Test) ^ this.GetHashCode(conditionalExpression.IfFalse) ^ this.GetHashCode(conditionalExpression.IfTrue);
MemberExpression memberExpression = obj as MemberExpression;
if (memberExpression.IsNotNull())
return (memberExpression.Member.IsNotNull() ? memberExpression.Member.GetHashCode() : -1) ^ this.GetHashCode(memberExpression.Expression);
ConstantExpression constantExpression = obj as ConstantExpression;
if (constantExpression.IsNotNull())
{
if (!constantExpression.Value.IsNotNull())
return -1;
return constantExpression.Value.GetHashCode();
}
ParameterExpression parameterExpression = obj as ParameterExpression;
if (parameterExpression.IsNotNull())
{
if (!parameterExpression.Name.IsNotNull())
return -1;
return parameterExpression.Name.GetHashCode();
}
NewExpression newExpression = obj as NewExpression;
if (newExpression.IsNotNull())
return this.GetArgumentsHashCode<Expression>(newExpression.Arguments) ^ (newExpression.Constructor.IsNotNull() ? newExpression.Constructor.GetHashCode() : -1);
return obj.ToString().GetHashCode();
}
private bool ArgumentsEquals<T>(ReadOnlyCollection<T> args1, ReadOnlyCollection<T> args2) where T : Expression
{
if (args1.Count != args2.Count)
return false;
for (int index = 0; index < args1.Count; ++index)
{
if (!this.Equals((Expression) args1[index], (Expression) args2[index]))
return false;
}
return true;
}
private int GetArgumentsHashCode<T>(ReadOnlyCollection<T> args) where T : Expression
{
return args.Aggregate<T, int>(0, (Func<int, T, int>)((hash, next) => hash ^ this.GetHashCode((Expression)next)));
}
}
}
We deployed the above code fix yesterday and have seen an insane improvement in CPU-usage and memory usage compared to before.
I've attached a graph of CPU usage and memory usage for our instances.
This is fixed in code. Now we're waiting for QA and then release.
Internal reference
FIND-6495 EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately
Bug not available on world yet.
We also noticed problem with concurrent dictionary - it takes a lot of memory. Is there any update in that case? Anybody can confirm that workaround provided by Martin works fine? Does it have any negative impact on functionality of epifind?
I can also confirm that fix proposed by Martin works correct, we have tested it on our production and our memory usage charts started looking as they should. Waiting for official fix from Episerver.
The bug fix is still under review, so it will be another two weeks at least to be released
Do we have any updates about the fix and release planning? Let us know how the review is going and share your expectations about potential release dates.
We can confirm that we are experiencing this problem in production environment aswell and are interesed in a official bugfix.
There have been difficulties with fixing the bug. If you can provide a memory dump that capture the issue, please send it to me (quan.mai@episerver.com) so we can analyse and work on an effective fix.
Any news on this one? I may have encontered this bug. I recently analyzed a memory dump, and there was an unreasonable ammount of objects related to Episerver Find queries (34 million). The Garbage Collector runs more and more often, causing tons of CPU usage and high response times. We have also expierenced instances just going down.
The bug has been fixed (i.e. reviewed and merged), but it will be some time before 13.2.6 gets out of the door (QA and everything)
Would you mind sending me the memory dump? Might be interesting looking into it
That's great news!
I would be happy to help with a memory dump. The dump is 4.8 GB (~ 400MB compressed). How shall we proceed?
Perhaps you can zip it and send it via wetransfer ? my email as is above - quan.mai [at] episerver.com. Thanks!
This is fixed as of Find 13.2.6
https://nuget.episerver.com/package/?id=EPiServer.Find
Changes
https://world.episerver.com/documentation/Release-Notes/?packageFilter=EPiServer.Find&typeFilter=All
The private static field EPiServer.Find.LambdaExpressionExtensions._cache keeps growing indefinately even when executing the same query and receiving the same result over and over. This will eventually lead to the worker process running out of memory (in our case, quite frequently).
Below is a code snippet that re-creates the behavior.
// The simplest possible query returning the same result every time will still
// increase the number of entries in the cache.
var cacheField = typeof(LambdaExpressionExtensions)
.GetField("_cache", BindingFlags.NonPublic | BindingFlags.Static);
var cache = cacheField.GetValue(null) as ConcurrentDictionary<Expression, Delegate>;
cache.Clear();
Debug.Assert(cache.Count == 0);
var findClient = ServiceLocator.Current.GetInstance<IClient>();
// Make the simplest possible query to return one hit.
var firstHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();
Debug.Assert(firstHits.Count == 1);
var firstHitID = firstHits[0].Id;
var firstHitCacheCount = cache.Count;
// The initial query should of course add entries to the cache. About 25 in my case.
Debug.Assert(firstHitCacheCount > 0);
// Make the same query in a loop.
for (var i = 0; i < 10; i++)
{
var loopedHits = findClient.UnifiedSearch().Take(1).GetResult().Hits.ToList();
Debug.Assert(loopedHits.Count == 1);
var loopedHitID = loopedHits[0].Id;
Debug.Assert(loopedHitID == firstHitID);
var loopedHitCacheCount = cache.Count;
// Given the same search query and the same result, the cache should not grow.
// It grows with each iteration.
Debug.Assert(loopedHitCacheCount <= firstHitCacheCount);
}