SaaS CMS has officially launched! Learn more now.

DXP - System.ComponentModel.Win32Exception (258) or Timeout period elapsed prior to obtaining a connection from the pool

Vote:
 

Here's a simple KQL query you can run in AI Logs to summarize exceptions from the Azure SQL calls:

dependencies
| where target contains ".database.windows.net"
| where tostring(customDimensions.Exception) !contains "Violation of"
| summarize sum(itemCount) by substring(tostring(customDimensions.Exception),0,275)
| order by sum_itemCount desc
| project sum_itemCount, Column1

Looking into stability issues and when problems arise we see large amounts of these:

  1. Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): Unknown error 258
  2. Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

I'm interested if others are also seeing these two exceptions in some volume for their CMS 12 on DXP?

Here's an issue for SqlClient that suggests this may be a broader issue with dotnet core running in Linux containers in general:
https://github.com/dotnet/SqlClient/issues/1530
 
For the DXP this suggests some tweaks are already in place:
https://support.optimizely.com/hc/en-us/articles/4432366206733-CMS-12-site-crash-due-to-SQL-timeout-error-when-working-in-CMS-edit-mode

#307196
Aug 21, 2023 7:24
Vote:
 

A shot in the dark but maybe this is with MARS enabled? When I was testing performance for CMS 12 MARS does not work well on Linux

#307293
Aug 22, 2023 14:00
Johan Kronberg - Aug 22, 2023 14:37
I got some info on that too; checked with operations and they set the connectionstring without the parameter in it (which should mean False).
We got better stability upgrading SqlClient to 5.1.1 (from 5.0.1 in our case). We are also now testing upgrade to .NET 7 from 6 on Integration to see if that gives us some improvements.
Quan Mai - Aug 22, 2023 14:53
interesting, keep us posted please
Johan Kronberg - Sep 11, 2023 17:54
Unfortunately we have logged occurrences of both exceptions also after going to .NET 7 in production.
Vote:
 

Hi Johan, did you manage to resolve these errors? We're receiving the same error in DXP 😞

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  
The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
The statement has been terminated.
System.ComponentModel.Win32Exception (258): Unknown error 258
#311918
Nov 03, 2023 4:58
Vote:
 

We still see them and usually at high volume in proximity of reported downtime. But there can also be tens of millions operations between showings.

Here is another issue with the client package that we are following since we usually see the problem around queries that are doing heavier work. Unfortunately they are not our own queries but if they were this issue thread suggest re-working those calls to be sync and not async.

https://github.com/dotnet/SqlClient/issues/593

#311920
Nov 03, 2023 8:15
Johan Kronberg - Nov 03, 2023 8:20
We recently went live with Microsoft.Data.SqlClient 5.1.2 and no errors reported yet but too early to say something definitive.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.