November Happy Hour will be moved to Thursday December 5th.

Improve error handling during Azure Service Bus partition errors

Found in

EPiServer.Azure 7.17.0

Fixed in

EPiServer.Azure 9.4.3

(Or a related package)

Created

Mar 05, 2018

Updated

Mar 12, 2018

Area

CMS Core

State

Closed, Fixed and tested


Description

Sometimes, the Service Bus reports that a subscription was successfully created. Then, when trying to use that subscription, it reports a "The remote server returned an error: (404) Not Found", and the subscription was never created.

The error is not transient and is not retried, causing the event system to stop working. The error can only happen if an instance starts up the first time and, at the same time, the Azure Service Bus experiences "partition errors". This fix retries MessagingEntityNotFoundException even though it is returned after the subscription has been created. The fix also

  • increases the buffer from 1,000 to 10,000 events (to withstand longer outages)
  • adds additional logging prefixed "Unexpected error" if it finds unexpected situations

EPiServer.Azure.Events.AzureEventProvider : Failed to initialize ServiceBus connection, no remote events will be sent or received until issue is solved and application is restarted
System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: The remote server returned an error: (404) Not Found. TrackingId:xxx, SystemTracker:xxx:MySiteEvents/Subscriptions/xxx, Timestamp:3/2/2018 12:11:16 AM ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.<>c.<GetAsyncSteps>b__22_1(GetAsyncResult`1 thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
— End of inner exception stack trace —
at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception)
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.EndGet[TEntityDescription](IAsyncResult asyncResult, String[]& resourceNames)
at Microsoft.ServiceBus.NamespaceManager.EndGetSubscription(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CompleteSynchronously[TResult](Task`1 task)
at EPiServer.Azure.Events.Internal.DefaultServiceBusSetup.CreateSubscription(String connectionString, String topicName, String subscriptionName, String filterProperty, CancellationToken cancellationToken)
at EPiServer.Azure.Events.AzureEventProvider.InitializeServiceBusConnection()
at System.Threading.Tasks.Task.Execute()