We recently had our very old corporate site rebuilt from the ground up. Old site was EPI CMS 6.X. New site is EPI 11.
Our production configuration is 2 web servers configured behind a load balancer.
Same code and configuration on both servers, deployed by our deployment system (Octopus)
The AppData folder in different location than default and replicated by DFS between the two servers for the blobs. The search index is also in this location and was DFS replicated but we stopped that thinking it may have been contributing to some of the problems.
We have been running into a lot of issues since moving to production (test environments were not load balanced). However this is the same load balance setup we had with the old site and didn't have these issues but that was a much older version of EPI.
The first issue is that content updates don't seem to be reflected. This is from our system engineer:
It appears that updates in the CMS are not immediately available on other web servers pulling content from the same database. The content becomes available once the application pool restarts. This seems to indicate that Epi does not automatically query for new content, but perhaps has a cached flag of some sort to indicate whether new content should be polled which is reset on application startup.
Secondly and more importantly the search index has been giving us periodic issues. We will get errors about it being corrupt and items left in queue. A few times it gets so bad the site becomes unresponsive and we have to shut it down, delete the index folder and then bring it back up. It also doesn't appear to be properly indexing all the content.
I've done a lot of googling and there doesn't seem to be a comprehensive guide for best practice load balanced setup. Can someone point me to such a guide if it exists?
To get the cache to invalidate correctly you need to configure the event system, see https://world.episerver.com/documentation/developer-guides/CMS/event-management/ and especially how to configure the built-in WCF provider. This should be fairly similar to how it worked in CMS 6 if I remember correctly.
Have you seen the documentation for the legacy search service (that you seem to be using) https://world.episerver.com/documentation/developer-guides/CMS/search/full-text-search-legacy/?
You are probably better off setting up the indexing and search service as a separate web app. Having two applications indexing against the same index is not supported I think.
The problem with with a seperate web app is that app would also be hosted on same 2 servers behind the load balancer so wouldn't the same problem exist with multiple apps trying to write to the shared index location?
In my googling many posts refer to this solution (for some reason I can't do hyperlinks here so adding spaces:
world. episerver. com/ blogs/ Henrik-Fransas/ Dates/ 2015/ 8/ setup-episerver-search-in-a-future-safe-and-stable-way/
The article has the following paragraph:
This is the basic way of configure EPiServer Search and it will work for small projects that are not in a load balanced environment or on Microsoft Azure. The problem with having this configuration in production in a load balanced environment is that if you put the index files on a common drive and let all the websites index then the index files will be corrupt. You can solve this by setting only one of the sites to do the index and that usually works, but I have had times when it fails because of for example problems with routing and so on.
However I cannot find any information on how to do the bolded piece. Just setup one of the two servers to do the indexing?
Odd this is a "legacy" search system. We had a third party build this site for us. Is there a non-legacy option they should have used?
Your're right, no need to have it as a separate web app.
When you configure the client there is a setting called readOnly that you can set to true on one of the webservers, https://world.episerver.com/documentation/developer-guides/CMS/search/full-text-search-legacy/Configuring-Optimizely-full-text-search-service/. Then only one of the clients will actually index content.
Optimizely provide search-as-a-service https://world.episerver.com/products/#Search. But the legacy search works for basic search capabilities, but as you've noticed, it has its shortcomings. You are not limited to use any of the search engines, you can integrate any search engine. But these two integrates well with the CMS out-of-the-box.
OK let's say one of the two servers is set to read-only for the index but both servers point to shared (or DFS replicated) index location.
If I make a content edit that would require an index update and it's initiated from the server that is read only how would the index be updated? Is that handled by the event management setup? Meaning does it tell the other server about the content edit and then that server tries to update index and because it's not read only it would be sucessful?
Same question for when just manually re-indexing through the Admin. Does that also event fire over to other server so that if kicked off from read only server the other server would get the event and take care of it?
I'm not that familiar with the legacy search, so not sure how the whole indexing process works. But I'm pretty sure there is a queue stored in the database which the indexing services is using. So regardles which server makes content changes, the indexing queue is updated. You probably have to validate this, by making a content update from both servers and then check if the index was updated.
Another solution is to just run the admin and editing UI on one of the servers.