November Happy Hour will be moved to Thursday December 5th.

AI OnAI Off

Connection Pool Issues Following Upgrade to EpiServer 11

Michael
Michael
Vote:
 

Hello,

We have recently upgraded our version of EpiServer CMS from 7.5 to 11. It's mostly working fine but occasioanlly we get an issue where the site will take up all available connections on the database server, causing not only EpiServer to stop working but any other site that might be using that database server.

An example of the error we get is below but I suspect the entry is a syptom of the issue rather than the cause and it is never preceded in the logs by anything consistant. We don't get this issue in EpiServer 7.5 and the only changes we've made during the upgrade are those required as part of the upgrade.

Has anyone else experienced this following an upgrade? Is there an easy way to identify what could be causing the site to be using so many connections (it's definetly our EpiServer site as we have checked the database)? I suspected one of the internal scheduled jobs but short of disabiling them all and then starting them one by one I'm not sure how to identify which one?

Many thanks!

2020-09-11 10:20:05,047 ERROR [33] EPiServer.Framework.Cache.ObjectInstanceCacheExtensions.ReadThroughWithWait - Failed to Read cacheKey = 'EPSegmentData:68175:xxxx'
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at EPiServer.Data.Providers.Internal.ConnectionContext.<OpenConnection>b__15_0()
   at EPiServer.Data.Providers.SqlTransientErrorsRetryPolicy.Execute[TResult](Func`1 method)
   at EPiServer.Data.Providers.Internal.ConnectionContext.OpenConnection()
   at EPiServer.Data.Providers.Internal.SqlDatabaseExecutor.GetConnection(Boolean requireTransaction)
   at EPiServer.Data.Providers.Internal.SqlDatabaseExecutor.<>c__DisplayClass31_0`1.<Execute>b__0()
   at EPiServer.Data.Providers.SqlTransientErrorsRetryPolicy.Execute[TResult](Func`1 method)
   at EPiServer.Core.ContentProvider.<>c__DisplayClass77_0.<GetMatchingSegments>b__0()
   at EPiServer.Framework.Cache.ObjectInstanceCacheExtensions.ReadThroughWithWait[T](IObjectInstanceCache cache, String cacheKey, Func`1 readValue, Func`2 evictionPolicy)
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at EPiServer.Data.Providers.Internal.ConnectionContext.<OpenConnection>b__15_0()
   at EPiServer.Data.Providers.SqlTransientErrorsRetryPolicy.Execute[TResult](Func`1 method)
   at EPiServer.Data.Providers.Internal.ConnectionContext.OpenConnection()
   at EPiServer.Data.Providers.Internal.SqlDatabaseExecutor.GetConnection(Boolean requireTransaction)
   at EPiServer.Data.Providers.Internal.SqlDatabaseExecutor.<>c__DisplayClass31_0`1.<Execute>b__0()
   at EPiServer.Data.Providers.SqlTransientErrorsRetryPolicy.Execute[TResult](Func`1 method)
   at EPiServer.Core.ContentProvider.<>c__DisplayClass77_0.<GetMatchingSegments>b__0()
   at EPiServer.Framework.Cache.ObjectInstanceCacheExtensions.ReadThroughWithWait[T](IObjectInstanceCache cache, String cacheKey, Func`1 readValue, Func`2 evictionPolicy)
#227742
Sep 11, 2020 9:52
Sanjay Kumar
Sanjay Kumar
Vote:
 

Hi Michael,

  1. The issue looks like the caching related, please check Episerver Caching Issue .Net 4.7 and try to override the default implementation of IObjectInstanceCache.
Failed to Read cacheKey = 'EPSegmentData:68175:xxxx'

   2. And try to increase Connection Timeout=30;

#227755
Sep 11, 2020 19:19
Michael
Michael
Vote:
 

Hi,

Thanks for the response. Unfortunatley your fix would not really solve the issue as the error shown was a symptom of what was going wrong rather than the cause of it (plus the bug you link to was fixed in CMS 10).

I have discovered this was an issue with one of our plugins/scheduled tasks which was making database calls within a Parallel.ForEach (bad!). This caused the code to fire off loads of simultaneous requests to the database and lock up the connection pool!

I discovered what was causing the issue by running a SQL Profile on the DB after firing the site up and tracing that back to the code.

I'm not sure why we only discovered this in testing but the job is scheduled to run at 5am (so I guess we never noticed) and I think the process of restoring DBs to the test environment meant the job had more rows to process than normal. 

#230072
Oct 29, 2020 9:41
This topic was created over six months ago and has been resolved. If you have a similar question, please create a new topic and refer to this one.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.