AI OnAI Off

EPiServer Events Initialization Issue - ASP.Net Compatibility

Cevin
Cevin
Vote:
 

Hi,

Having a pretty major problem with the EpiServer Events Service and cannot get to the bottom of it...

This problem is causing pretty major stability issues on the site. Running two IIS servers (V8.5) in load a balanced environment, with events provider configured over TCP. 

The configuration is the same on both servers, as follows:

The site is running fine most of the time but occasionally the events service seems to encounter problems shutting down and failing to restart!

Receiving exceptions relating to the event service being unavailable and messges being dropped from server A. The events service has died/does not start on server B. The application cannot be started with errors on initialization relating to the ASP.Net compatibility mode. Eventually does start and can be brought back online.

The error has happened on both servers at different times.

Can anyone help on understanding why takes happens? Or perhaps problems with the configuration?

Any help would be greatly appreciated...

Regards,
Cevin

The stack trace from the unhandled async exeption is below...

Exception information: 
    Exception type: TargetInvocationException 
    Exception message: Exception has been thrown by the target of an invocation.
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Exception has been thrown by the target of an invocation.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at EPiServer.Framework.Initialization.InitializationEngine.OnInitComplete()
   at EPiServer.Framework.Initialization.InitializationEngine.ExecuteTransition(Boolean continueTransitions)
   at EPiServer.Framework.Initialization.InitializationModule.EngineExecute(HostType hostType, Action`1 engineAction)
   at EPiServer.Framework.Initialization.InitializationModule.FrameworkInitialization(HostType hostType)
   at EPiServer.Global..ctor()
   at BLANK.EPiServerApplication..ctor()
   at ASP.global_asax..ctor()
One or more errors occurred.
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at EPiServer.Events.EventsInitialization.<>c__DisplayClass6_0.b__1(Object sender, EventArgs args)
One or more errors occurred.
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
   at EPiServer.Events.EventsInitialization.InitializeEvents(ISiteSecretManager siteSecretManager, IEventSiteRegistry eventSiteRegistry, EventProviderService eventProviderService)
   at System.Threading.Tasks.Task.Execute()
The service cannot be activated because it does not support ASP.NET compatibility. ASP.NET compatibility is enabled for this application. Turn off ASP.NET compatibility mode in the web.config or add the AspNetCompatibilityRequirements attribute to the service type with RequirementsMode setting as 'Allowed' or 'Required'.
   at System.ServiceModel.Activation.HostedAspNetEnvironment.ValidateCompatibilityRequirements(AspNetCompatibilityRequirementsMode compatibilityMode)
   at System.ServiceModel.Description.DispatcherBuilder.ValidateDescription(ServiceDescription description, ServiceHostBase serviceHost)
   at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
   at System.ServiceModel.ServiceHostBase.InitializeRuntime()
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at EPiServer.Events.Providers.WcfEventProvider.CreateEventServiceHost()
   at EPiServer.Events.Providers.WcfEventProvider.b__13_0()
   at System.Threading.Tasks.Task.Execute()

[Pasting files is not allowed]

#186816
Jan 04, 2018 18:25
Erik Norberg
Erik Norberg
Vote:
 

I remember seeing it before but I don't know if we ever got any wiser on the cause of it.

We had major issues with it when performing deploys as it seemed that after restart it tried to sync with the other server which took a hit and pretty much became unresponsive.

We had three servers and we ended up with a cascading effect where we had to juggle the servers as they became none-responsive one-by-one and it could take over an hour until everything settled down and all servers was stabile. Nightly/scheduled iis-resets could trigger the cascading to start anew. cry

I don't really have a solution for you, only a recommendation:

Setup a new environment where the servers are on an isolated subnet and configure remote events to use udp-multicast instead of tcp. We never seen the issue since we made that move and the servers are stabile and reliable.

#186844
Jan 05, 2018 14:52
Cevin
Cevin
Vote:
 

Thanks @Erik. Yes its very frustrating and quite a strange issue :( ... onwards the troubleshooting goes, may have to try configuring UDP as you mentioned.

#186896
Jan 08, 2018 13:34
MMihaylov
MMihaylov
Vote:
 

We have exactly the same issue, for quite a lot of time. We use load balanced environment with 2 nodes of EpiServer. However, our configuration differs a little bit.

Node 1

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
  <services>
	<service name="EPiServer.Events.Remote.EventReplication">
	  <endpoint name="RemoteEventServiceEndPoint" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" bindingConfiguration="RemoteEventsBinding" address="net.tcp://hostname-01:1337/RemoteEventService" />
	</service>
  </services>
  <client>
	<endpoint name="RemoteEventServiceClientEndPoint" address="net.tcp://hostname-02:1337/RemoteEventService" binding="netTcpBinding" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" />
  </client>      
  <bindings>
	<netTcpBinding>
	  <binding name="RemoteEventsBinding" portSharingEnabled="true">
		<security mode="None" />
	  </binding>
	</netTcpBinding>        
  </bindings>
</system.serviceModel>



Node 2

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
  <services>
	<service name="EPiServer.Events.Remote.EventReplication">
	  <endpoint name="RemoteEventServiceEndPoint" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" bindingConfiguration="RemoteEventsBinding" address="net.tcp://hostname-02:1337/RemoteEventService" />
	</service>
  </services>
  <client>
	 <endpoint name="RemoteEventServiceClientEndPoint" address="net.tcp://hostname-01:1337/RemoteEventService" binding="netTcpBinding" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" />
  </client>
  <bindings>
	<netTcpBinding>
	  <binding name="RemoteEventsBinding" portSharingEnabled="true">
		<security mode="None" />
	  </binding>
	</netTcpBinding>       
  </bindings>
</system.serviceModel>



We have noticed that the issue happens after we deploy a new version. I guess after that some kind of initialization happens, causing the issue. Any help/advice would be much appreciated.

#187031
Edited, Jan 10, 2018 15:41
Cevin
Cevin
Vote:
 
<p>Hi @<a href="/System/Users-and-profiles/Community-Profile-Card/?userid=8df9bed7-1bf8-e511-9afb-0050568d2da8">MMihaylov</a>&nbsp;- I had tried this configuration&nbsp; (list hosted service for A and list client for B - and vice versa). This still did not fix the problem. There isn't much detail out there to understand how the service works, to help craft a correct config, pretty painful.</p> <p>I think the problem was being trigged because of some publishing events on startup - either this or some sequence for initalization between the two nodes underneath the hood of EpiServer. I played with this for some time with different conifigurations for TCP but nothing was reliable - this problem can really cause some nasty&nbsp;behaviour on initialization.</p> <p>We switched to using UDP (the nodes were already in their own subnet) with a config similar to that on&nbsp;<a href="/documentation/Items/Developers-Guide/Episerver-CMS/75/Event-management/WCF-event-management/">https://world.episerver.com/documentation/Items/Developers-Guide/Episerver-CMS/75/Event-management/WCF-event-management/</a>&nbsp;... this seems to have worked (for @Erik too) so I would recommend that.</p> <p></p> <p>Good luck</p>
#187032
Jan 10, 2018 15:57
MMihaylov
MMihaylov
Vote:
 

Thanks @Cevin, 

at least I know that UDP configuration works. For now we need to stick to TCP. If I find something else, I will update the thread. 

#187037
Edited, Jan 10, 2018 16:03
This topic was created over six months ago and has been resolved. If you have a similar question, please create a new topic and refer to this one.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.