Syncronization of products to multiple web fronts

Vote:
 

We have one editor node with episerver commerce v.12.12 installed. On that node we have no problem to browse directly to any products. We are experiencing problems with the syncronization to our web fronts. When I edit a product in epi commerce, for some products, they give me a Http 404 when browsing to them on the web fronts. Sometime it works on one of the web fronts, and sometimes it doesn't work on any. After some time, up to a couple of days, the products are browsable on the site again. I don't know how to troubleshoot this.

Please if you can give me some advice I would be really greatfull !

//Daniel

#249387
Edited, Mar 01, 2021 10:17
Vote:
 

Hi Daniel,

That sounds a lot like an issue with remote events between the edit node and the public facing nodes. It sounds like the change is being made but the event isn't reaching the web frontends and hence the cache is only refreshing when it expires (or the app pool recycles). How are your remote events configured? UDP? Do you have the same issue with CMS content or is it just commerce?

#249392
Mar 01, 2021 12:04
Vote:
 

Second what Paul said, sounds like a problem with remote cache invalidation. Are you on Azure/DXP? If yes you should have your ServiceBus endpoint configuration checked. 

#249393
Mar 01, 2021 12:08
Vote:
 

Hi guys! Thanks for your quick replies. We are not in the cloud. I will look into if CMS content is coming over, and get back with my findings.

#249399
Mar 01, 2021 12:59
Vote:
 

We do not experience the same problem with the CMS syncronization as far as we know.

#249401
Mar 01, 2021 13:28
Vote:
 

If it is on premise you might also want to ask your IT to look into firewall settings.

What puzzles me is that you said it might take a day for content to appear. That does not sound right (the default cache time out is about 10m)

#249403
Mar 01, 2021 13:51
Daniel Sandberg - Mar 03, 2021 7:41
Yes. Sometimes it gets updated right away, but sometimes it gets updated on some of our servers, but takes a couple of days before every server has the new information.
Vote:
 

this is how our confguration looks like: 

    <client>
      <!-- Remote Events --> 
      <endpoint name="RemoteEventServiceClientEndPoint" xdt:Locator="Match(name)" xdt:Transform="Remove" />
      <endpoint name="mysite.se-10.X.X.X" address="net.tcp://10..X.X.X:Y/RemoteEventService" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" xdt:Transform="Insert" />
      <endpoint name="mysite.se-10.X.X.X" address="net.tcp://10..X.X.X:Y/RemoteEventService" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" xdt:Transform="Insert" />
      <endpoint name="mysite.se-10.X.X.X" address="net.tcp://10..X.X.X:Y/RemoteEventService" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" xdt:Transform="Insert" />
      <endpoint name="mysite.se-10.X.X.X" address="net.tcp://10..X.X.X:Y/RemoteEventService" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" binding="netTcpBinding" xdt:Transform="Insert" />
    </client>
#249405
Mar 01, 2021 14:14
Vote:
 

Where is the caching for the syncronization to be found? 

#249406
Mar 01, 2021 14:21
Vote:
 

I think also this is relvant. looks like communication on UDP.

    <services>
      <service name="EPiServer.Events.Remote.EventReplication" xdt:Locator="Match(name)" xdt:Transform="Replace">
        <endpoint name="RemoteEventServiceEndPoint" contract="EPiServer.Events.ServiceModel.IEventReplication" bindingConfiguration="RemoteEventsBinding" address="soap.udp://X.X.X.X:5000/RemoteEventService" binding="customBinding" />
      </service>
    </services>
    <client>
      <endpoint name="RemoteEventServiceClientEndPoint" address="soap.udp://X.X.X.X:5000/RemoteEventService" binding="customBinding" bindingConfiguration="RemoteEventsBinding" contract="EPiServer.Events.ServiceModel.IEventReplication" xdt:Locator="Match(name)" xdt:Transform="Replace" />
    </client>
    <bindings>
      <netTcpBinding xdt:Transform="Remove" />
    </bindings>
#249466
Mar 02, 2021 12:10
Vote:
 

I haven't found any loggins for this remote event update. Should there be any?

#249503
Mar 03, 2021 7:42
Vote:
 

I found this in the logs:

Error creating/opening service host for event replication class System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint X.X.X.X:5000. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations
#249548
Edited, Mar 04, 2021 7:47
Vote:
 

that sounds suspicious, if you have access to the server, run netstat -aon | findstr '[port_number]' command to see which program is using that port

#249552
Mar 04, 2021 9:07
Daniel Sandberg - Mar 04, 2021 15:39
I see that localhost is listening, and the other web fronts are established.
Daniel Sandberg - Mar 04, 2021 15:47
I am thinking maybe I need to set this portSharingEnabled="true", because current implementation lacks that.
Vote:
 

As is today, I still experience the same problem. If I restart one of the web fronts that doesn't show the product in IIS it starts working again. So this has to do with the caching right?

#250249
Mar 16, 2021 7:56
Vote:
 

Yes it is most likely it. It seems that instance had some issues which prevents it from receiving or processing the cache invalidation messages. Now you restart it (the cure for any problem!) it started working again

#250251
Mar 16, 2021 8:58
Vote:
 

Problem is, it happens every now and then. So it is not a single incident. We cannot obviously restart the site everytime we update a product.

#250293
Mar 17, 2021 7:33
Vote:
 

I'd agree rrestarting IIS after each publish isn't the most practical of approaches, even if it does seem to work.

It certainly sounds like it's an issue with the cache not invalidating when it should, probably due to an issue with remote events. From your posts above, it's not clear what mechanism you're using for remote events. In one snippet you show remote events configured using TCP and in another it's using UDP. First of all, are all servers (both front-end and CMS) set up to use the same mechanism? If so, are all servers in the same network with no devices such as firewalls which would block traffic between the CMS and the front-end servers? UDP can be particularly fiddly to set up and debug as it tends to be blocked on networks plus, being UDP, there are no guarantees of delivery.

It might be worth temporarily installing the Episerver developer tools add on as I believe that has tools for testing remote events. That said, it may be of limited use if you can't access the tool on each of the boxes.

It may also be worth considering switching the remote events mechanism to one of the alternatives, even if it's just temporarily to test whether it resolves the issue. Probably the easiest to set up is Azure service bus which works even if your servers aren't in Azure.

#250780
Mar 17, 2021 19:18
Daniel Sandberg - Mar 30, 2021 7:01
If communication is blocked by firewalls and such, would error be logged in Event Viewer or similar? I feel this is kind of searching in the dark.
Daniel Sandberg - Mar 30, 2021 7:03
You mention the Azure service bus. Sounds to me like a big implementation change? How could it be the easiest?
Daniel Sandberg - Mar 30, 2021 7:15
Regarding firewalls. The products are after a couple of hours reachable also at the web fronts. This implies the products gets through, but the cache is not invalidated?
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.