Release notes for Optimizely CMS and Customized Commerce updates

When CMS starts and fails early, the Azure ServiceBus provider throws a NullReferenceException in Uninitialize.

Warning can be ignored.

[15:02:25 WRN] Failed to stop processing subscriptions
System.NullReferenceException: Object reference not set to an instance of an object.
   at EPiServer.Azure.Events.Internal.DefaultServiceBusEventProvider.Uninitialize() in E:\BuildAgent2\work\edfe936b4017877a\src\EPiServer.Azure\Events\Internal\DefaultServiceBusEventProvider.cs:line 435 

Exception when trying to start message pump in Service Bus. The error indicates that the just created subscription is not available. Operation should be retried.

2019-10-23T09:07:49,Error,xxx,4741f2,637074184691094985,0,76,27,"EPiServer.Azure.Events.AzureEventProvider : Failed to initialize ServiceBus connection, no remote events will be sent or received until issue is solved and application is restarted
System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: 40400: Endpoint not found., Resource:sb://xx.servicebus.windows.net/mysitevents/subscriptions/xx. TrackingId:6db53c41-1745-4e18-8fff-e66c4d599072_G42, xx.servicebus.windows.net:mysitevents/Subscriptions/xx, Timestamp:2019-10-23T09:07:48 ---> System.ServiceModel.FaultException: 40400: Endpoint not found., Resource:sb://xxx.servicebus.windows.net/mysitevents/subscriptions/xx. TrackingId:6db53c41-1745-4e18-8fff-e66c4d599072_G42, xxx.servicebus.windows.net:mysitevents/Subscriptions/xx, Timestamp:2019-10-23T09:07:48
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_3(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass8_1.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_3(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.EndReceiveCommand(IAsyncResult result, IEnumerable`1& messages)
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Common.ExceptionExtensions.ThrowException(Exception exception)
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.MessageReceiver.OnMessage(MessageReceivePump pump)
   at Microsoft.ServiceBus.Messaging.SubscriptionClient.OnMessage(Action`1 callback, OnMessageOptions onMessageOptions)
   at EPiServer.Azure.Events.AzureEventProvider.InitializeServiceBusConnection()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) 40400: Endpoint not found., Resource:sb://xx.servicebus.windows.net/mysitevents/subscriptions/xx. TrackingId:6db53c41-1745-4e18-8fff-e66c4d599072_G42, SystemTracker:xxx.servicebus.windows.net:mysitevents/Subscriptions/xx, Timestamp:2019-10-23T09:07:48<---

Steps to reproduce

1. Create Service Bus Premium.
2. Start the site.

Expected:
Should work.

Actual:
Throws an exception.

Workaround: Create a topic before starting the site, for example in an Azure portal.

System.ArgumentException: The remote server returned an error: (400) Bad Request. SubCode=40000. The property 'EnableExpress' cannot be set when creating a Topic because the namespace 'xxx' is using 'Premium' tier. To know more visit https://aka.ms/sbResourceMgrExceptions
Parameter name: EnableExpress. TrackingId:?????, xxx.servicebus.windows.net:???????, Timestamp:2019-12-05T17:15:25 ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.CreateOrUpdateTask`1.<OnRunAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.RetryResourceTask`1.<RunAsync>d__34.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.RetryResourceTask`1.<RunAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceBus.NamespaceManager.CreateOrUpdateTopicTask.<RunAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceBus.Common.TaskHelpers.CompleteSynchronously[TResult](Task`1 task)
   at EPiServer.Azure.Events.Internal.DefaultServiceBusSetup.CreateTopic(String connectionString, String topicName, Boolean enablePartitioning, Boolean enableExpressQueues, CancellationToken cancellationToken)
   at EPiServer.Azure.Events.AzureEventProvider.InitializeServiceBusConnection()

Newer versions of WindowsAzure.ServiceBus should be tested and allowed. Previously, only versions < 5.0 are allowed. As of this bug fix, the version must be 6.0.0 or higher.

The default was changed from 10 to 100, making consumption of events faster. In testing, we saw about a 6x performance improvement.
This bug fix also introduces a setting for the event provider: prefetchCount.

If code tries to send a large number of events through the Azure Event provider over a short time period, a large number of parallel threads are created. This might trigger throttling in Azure Service Bus, causing even more parallel threads to be created to handle new incoming events while previous events are being retried.

The Azure Event provider is supposed to limit the number of threads, and queue incoming events while existing events are being processed. The logic was not working as expected. It is now fixed.

Sometimes, the Service Bus reports that a subscription was successfully created. Then, when trying to use that subscription, it reports a "The remote server returned an error: (404) Not Found", and the subscription was never created.

The error is not transient and is not retried, causing the event system to stop working. The error can only happen if an instance starts up the first time and, at the same time, the Azure Service Bus experiences "partition errors". This fix retries MessagingEntityNotFoundException even though it is returned after the subscription has been created. The fix also

• increases the buffer from 1,000 to 10,000 events (to withstand longer outages)
• adds additional logging prefixed "Unexpected error" if it finds unexpected situations

EPiServer.Azure.Events.AzureEventProvider : Failed to initialize ServiceBus connection, no remote events will be sent or received until issue is solved and application is restarted
System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: The remote server returned an error: (404) Not Found. TrackingId:xxx, SystemTracker:xxx:MySiteEvents/Subscriptions/xxx, Timestamp:3/2/2018 12:11:16 AM ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.<>c.<GetAsyncSteps>b__22_1(GetAsyncResult`1 thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
— End of inner exception stack trace —
at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception)
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.EndGet[TEntityDescription](IAsyncResult asyncResult, String[]& resourceNames)
at Microsoft.ServiceBus.NamespaceManager.EndGetSubscription(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CompleteSynchronously[TResult](Task`1 task)
at EPiServer.Azure.Events.Internal.DefaultServiceBusSetup.CreateSubscription(String connectionString, String topicName, String subscriptionName, String filterProperty, CancellationToken cancellationToken)
at EPiServer.Azure.Events.AzureEventProvider.InitializeServiceBusConnection()
at System.Threading.Tasks.Task.Execute()

Sometimes, the Service Bus reports that a subscription was successfully created. Then, when trying to use that subscription, it reports a "The remote server returned an error: (404) Not Found", and the subscription was never created.
The error is not transient and is not retried, causing the event system to stop working.

The error can only happen if an instance starts up the first time and, at the same time, the Azure Service Bus experiences "partition errors". This fix retries MessagingEntityNotFoundException even though it is returned after the subscription has been created. The fix also

• increases the buffer from 1,000 to 10,000 events (to withstand longer outages)
• adds additional logging prefixed "Unexpected error" if it finds unexpected situations

EPiServer.Azure.Events.AzureEventProvider : Failed to initialize ServiceBus connection, no remote events will be sent or received until issue is solved and application is restarted
System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: The remote server returned an error: (404) Not Found. TrackingId:xxx, SystemTracker:xxx:MySiteEvents/Subscriptions/xxx, Timestamp:3/2/2018 12:11:16 AM ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.<>c.<GetAsyncSteps>b__22_1(GetAsyncResult`1 thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
— End of inner exception stack trace —
at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception)
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.EndGet[TEntityDescription](IAsyncResult asyncResult, String[]& resourceNames)
at Microsoft.ServiceBus.NamespaceManager.EndGetSubscription(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CompleteSynchronously[TResult](Task`1 task)
at EPiServer.Azure.Events.Internal.DefaultServiceBusSetup.CreateSubscription(String connectionString, String topicName, String subscriptionName, String filterProperty, CancellationToken cancellationToken)
at EPiServer.Azure.Events.AzureEventProvider.InitializeServiceBusConnection()
at System.Threading.Tasks.Task.Execute()

When an Alloy site on CMS 11.3 is setup in a DXC environment with failover, the failover instance is throwing an exception because it tries to write to a read-only blob replica.

Note: Caused by breaking change in NuGet dependency WindowsAzure.Storage 8.x.

Exception

Server Error in '/' Application.
The remote server returned an error: (403) Forbidden.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.WebException: The remote server returned an error: (403) Forbidden.

Stack Trace:

[WebException: The remote server returned an error: (403) Forbidden.]
System.Net.HttpWebRequest.GetResponse() +2452
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync(RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:677

[StorageException: The remote server returned an error: (403) Forbidden.]
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync(RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:604
Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(BlobContainerPublicAccessType accessType, BlobRequestOptions requestOptions, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlobContainer.cs:233
EPiServer.Azure.Blobs.DefaultAzureBlobContainer.CreateIfNotExist() +281
System.Threading.Tasks.Task.Execute() +73

[AggregateException: One or more errors occurred.]
System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken) +1307
System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout) +45
System.Threading.Tasks.Task.WaitAll(Task[] tasks) +28

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +221
System.Delegate.DynamicInvokeImpl(Object[] args) +160
EPiServer.Framework.Initialization.InitializationEngine.OnInitComplete() +320
EPiServer.Framework.Initialization.InitializationEngine.ExecuteTransition(Boolean continueTransitions) +232
EPiServer.Framework.Initialization.InitializationModule.EngineExecute(HostType hostType, Action`1 engineAction) +420
EPiServer.Framework.Initialization.InitializationModule.FrameworkInitialization(HostType hostType) +167
EPiServer.Global..ctor() +83
ASP.global_asax..ctor() +10

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck) +0
System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) +197
System.Activator.CreateInstance(Type type, Boolean nonPublic) +114
System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark) +1514
System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes) +300
System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture) +27
System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args) +80
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +225
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +318

[HttpException (0x80004005): Exception has been thrown by the target of an invocation.]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +405
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +113
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +604

Get-childItem : Cannot find path 'C:\EPiServer.Reference.Commerce-11.5.0-pre-002801\packages\EPiServer.Azure.9.4.0\lib\net461' because it does not exist.
At C:\EPiServer.Reference.Commerce-11.5.0-pre-002801\packages\EPiServer.Azure.9.4.0\tools\Install.ps1:124 char:1
+ get-childItem "$libPath*.dll" | % { AddOrUpdateBindingRedirect $_  $assemblyCon ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\EPiServer.Re....4.0\lib\net461:String) [Get-ChildItem], ItemNotFoundException