Event Manager crashes during product import

Tommy Ivarsson

Vote:

Out setup uses 1 admin server, 2 webfronts and 1 integration server. These all share 1 database. Remote events are activated and functioning, using TCP.

We run a lot of product imports. They write to the Commerce Catalog. This can be several thousand products to tens of thousands of products.

Since the site has gone live we have noticed that the jobs often fail with the following stack trace:

System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: The event manager has been stopped.
  at EPiServer.Events.ChangeNotification.Implementation.ManagedChangeProcessor`1.ConsumeNotifyChangeResults(IEnumerable`1 items)
  at EPiServer.Events.ChangeNotification.Implementation.ChangeRouter.NotifyChanges(IEnumerable`1 changes)
  at EPiServer.Events.ChangeNotification.Implementation.ChangeNotificationManager.NotifyChanges[TChangedData](IEnumerable`1 changes)
  at Mediachase.Data.Provider.TransactionScope.ExecuteActions(IEnumerable`1 actions)
  --- End of inner exception stack trace ---

This leads me to believe that we are overloading the Event Management System.

Once this happens the whole site dies and restarts.

Any help would be greatly appreciated. I would gladly provide any further information if such is needed.

#140469

Oct 20, 2015 22:53

Quan Mai

Vote:

Hi,

Do you write your custom code to import products?

Do you have custom listeners to the events?

#140479

Oct 21, 2015 8:42

Tommy Ivarsson

Vote:

Thanks a lot for your reply.

We have written a custom scheduled job which reads XML and uses the Commerce API to update or insert products.

We don't have any custom listeners listening to any events concerning product inserts or updates. We do however have a few custom event listeners which are used for logging and updating the Status Message. We have removed those now and will see if we have any more luck running the job now.

If the problems continue, is there any way to temporarily turn off the event publishing from this job and what type of consequences could that have?

#140483

Oct 21, 2015 10:09

K Khan

Vote:

You can comment these lines from your web.config if you dont want to use Event Listeneres

<services>
      <service name="EPiServer.Events.Remote.EventReplication" behaviorConfiguration="DebugServiceBehaviour">
        <endpoint name="RemoteEventServiceEndPoint" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="customBinding" bindingConfiguration="RemoteEventsBinding" address="soap.udp://239.255.255.19:5000/RemoteEventService" />
      </service>
    </services>
    <client>
      
      <endpoint name="RemoteEventServiceClientEndPoint" address="soap.udp://239.255.255.19:5000/RemoteEventService" binding="customBinding" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" />
    </client>

Regards

#140493

Oct 21, 2015 11:23

Tommy Ivarsson

Vote:

Thanks but that't not really the dynamic solution I was looking for. Doing that would require me to constantly update the configs manually.

#140497

Oct 21, 2015 11:33

Quan Mai

Vote:

@Tommy: Disabling event publishing will disable the cache synchronization across site instances, so use it with great caution.

Does your code update the products in a parallel manner somehow? We tested with some very big catalog and the importing with CatalogImportExport went fine - of course our test environment might not be as complex as your production one.

#140498

Oct 21, 2015 11:43

Tommy Ivarsson

Vote:

We are not updating products in a parallel manner. We are however getting errors like this all the time on our webfronts, often followed by that the site restarts.

System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint 10.60.152.21:13002. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations. ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at System.ServiceModel.Channels.SocketConnectionListener.Listen()
   --- End of inner exception stack trace ---
   at System.ServiceModel.Channels.SocketConnectionListener.Listen()
   at System.ServiceModel.Channels.ConnectionAcceptor.StartAccepting()
   at System.ServiceModel.Channels.ExclusiveTcpTransportManager.OnOpen()
   at System.ServiceModel.Channels.TransportManager.Open(TransportChannelListener channelListener)
   at System.ServiceModel.Channels.TransportManagerContainer.Open(SelectTransportManagersCallback selectTransportManagerCallback)
   at System.ServiceModel.Channels.TcpChannelListener`2.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at EPiServer.Events.Providers.WcfEventProvider.CreateEventServiceHost()

Perhaps there is a connection between the two.

#140501

Edited, Oct 21, 2015 12:39

K Khan

Vote:

Can you share your Complete Configuration Node
Regards
/K

#140550

Oct 22, 2015 15:04

Tommy Ivarsson

Vote:

Do you want the full web.config or just the parts configuring remote events? Would you like them for all 4 servers or just for one? If one, which?

#140552

Oct 22, 2015 15:58

K Khan

Vote:

just the parts remote events configurations please

#140554

Oct 22, 2015 16:13

Tommy Ivarsson

Vote:

Thanks a lot for looking at this.

These are excerpts from 4 different configs; Integration, Admin, Web1 and Web2.

I tried pasting them straight into the post but I am getting told that my posts have been flagged as spam, so here it is on Pastie instead:

http://pastie.org/10499925

#140560

Edited, Oct 22, 2015 16:48

K Khan

Vote:

May be I am overthinking,

what will be result if you will Comment below client from Admin

#140568

Oct 22, 2015 18:29

Tommy Ivarsson

Vote:

We now seem to have got this issue under control.

We realised that there were a lot of changes to the site root folder on IIS. These changes were mostly caused by having our ImageResizer disk cache-folder in the site folder. Moving it to a different directory has pretty much killed the issue.

Thanks everyone for your help!

#141266

Nov 10, 2015 16:11

Dan Valea

Vote:

I'm having the same problem and under the same circumstance; we are running a Job that migrates the content of products into EPiServer DB and for some reason we receive the "The event manager has been stopped." error. I tried to remove that configuration section @K Khan was mentioning but nothing changed, the error is still there.

Also, I think is worth mentioning that the application is hosted in an Azure environment configured by EPiServer.

Any hints on this would be much appreciated.

Thank you!

#179626

Jun 16, 2017 16:25