harry mohammed
Feb 26, 2013
  1465
(2 votes)

CMS editor server instance crashing main site

 

Recently I was involved in supporting a new large scale implementation for a global client.

The implementation had gone through full testing and passed with flying colours. There seemed to be the usual cosmetic bugs and certain “Known bugs” that were acceptable by the client.

We finally got the go ahead to switch the new site on, this is where the real fun started.

Once the Load Balancer changes were made and the site made publicly available, we saw issues which were causing key functionality to have huge performance issues and ultimately crashing the site. When I say crashing the site, I mean the servers stopped responding.

The configuration of the implementation was to have the following:

  • 1 CMS editor server
  • Front end presentation servers with CMS access switched off.
  • Akamai CDN

It seemed a standard set up and nothing new.

What we found, was Cached objects were being invalidated for data sources from a different provider. After a while the front end presentation servers would stop responding and cause a failover.

Initial investigation was made, however, this was inconclusive and was pointing at Akamai not caching correctly.

The goose chase started and a series of remedial steps were taken:

  1. Monitor the DB
  2. Look at deactivating the many languages.

Monitoring the DB showed masses of Deadlocks and the dreaded SQL Query that is initiated by FindPagesWithCriteria were causing the DB to lock up.

However, this was not actually the issue.

If we think about it, the CMS editor application was being bought up causing the Presentation servers and DB to max out.

Further thinking it seems logical to think that the CMS was causing the issue.

To prove the point, we stopped the CMS server and the issue went away immediately and the Site was responding as expected.

After looking at the implementation it was identified that the Remote Events that are standard EPiServer built in functionality and I have never seen cause an issue.

What was realised was that the CMS and front end instances were configured to listen to sites that were not actually active. (There were 2 other sites that weren’t actually switched on)

So what was happening, the CMS was sending out events to all the presentation servers announcing it self and trying to register itself which was causing the Presentation servers to invalidate their cache. With users still trying to browse the site, this was causing them all to go back to populate their caches which cause the DB to max out and causing the sites to crash.

To fix this, the following is what to do:

  1. Set the scheduler to false for all servers and all sites in episerver.config.
  2. If other sites are not running in ISS make sure to comment them out in the episerver.config sites section.
  3. Delete the content of automaticSiteMapping section, an attribute in episerverFramework.config on all server. Its recreated automatically on site startup.
  4. Delete table tblsiteConfig, recreated on startup
  5. Make sure no scheduled jobs are active in admin mode.
  6. Restart frontend sites.

 

The point of the above is, when switching a site to live, make sure only that is running what is needed.

Feb 26, 2013

Comments

Please login to comment.
Latest blogs
Display Child Pages in Content Delivery API Response

The below example will implement an instance of IContentConverterProvider to customise the serialisation of PageData and output child pages in the...

Minesh Shah (Netcel) | Oct 4, 2022

Bring the Report Center back in Optimizely CMS 12

The Report Center has been a part of Optimizely CMS since its first debut in version 5R2 in 2008, but in CMS 12, it's removed! Don't despair! Make...

Tomas Hensrud Gulla | Oct 4, 2022 | Syndicated blog

Customizing Property Lists in Optimizely CMS

Generic property lists is a cool editorial feature that has gained a lot of popularity - in spite of still being unsupported (officially). But if y...

Allan Thraen | Oct 2, 2022 | Syndicated blog

Optimizely names Luminary Senior Developer, Ynze Nunnink, OMVP

Luminary Senior Developer and Optimizely Lead, Ynze Nunnink has secured the coveted position of Optimizely MVP. Earning a Platinum badge for his...

Ynze | Oct 2, 2022 | Syndicated blog

Content Delivery API – The Case of the Duplicate API Refresh Token

Creating a custom refresh provider to resolve the issues with duplicate tokens in the DXC The post Content Delivery API – The Case of the Duplicate...

David Lewis | Sep 29, 2022 | Syndicated blog

A multi-brand strategy in Optimizely DXP

The Optimizely Digital Experience Platform (DXP) is ideally suited to digital projects featuring multiple websites or applications. Here, we showca...

Ynze | Sep 29, 2022 | Syndicated blog