Deployment in DXC affects the Prod site before going live

Georg Sandholm

Vote:

Hi,

We are managing a quite transaction intensive Episerver Commerce site that is running on DXC. For smaller changes, without DB schema or content type updates, we do our deploys without maintenance page. It works without errors, but we experience very slow performance on the live site during deploy to the slot before we hit "Go live". It seems that the deploy to the slots affects the live environment heavily. Our first thought was that this might be because of the dependency on the same database. But we cannot see that the database is anywhere near 100%.

Anyone experiencing the same problem? Do you have any ideas what the cause of this could be?

/Georg

#227908

Sep 16, 2020 6:57

Paul Gruffydd

Vote:

Hi Georg,

It's not something I've noticed but I wonder whether it's resource contention between the slot and the production site. The deployment slot on an azure app shares the same resources as the public-facing site and starting up an Episerver site, particularly a commerce site, can be quite resource intensive. I suspect that warming up the site on the slot is tying up resources that would otherwise be used by the production site.

Do you see anything useful in application insights during those periods of slow performance?

#228048

Sep 18, 2020 7:46

Georg Sandholm

Vote:

Hi Paul,

Thank you for your answer. I know that the slot and the live web app is running on the same service plan. But when I look in App Insights I would expect to see something like near 100% CPU or near 100% memory, but this doesn't seem to be the case. The databases also looks fine, well below their capacity. It looks like something in the slot is locking, or really slowing down, the live app. But it is really hard to analyze this without knowing in detail what happens during a deploy.

This is a real problem for us, because it prohibits even the smallest changes from being deployed during daytime.

#228109

Sep 18, 2020 12:13

Stefan Holm Olsen

Vote:

Hi Georg

I would investigate whether your cache entries are getting cleared when you warm up the deployment slot.

Because both the live slot and the deployment slot are sharing the same ServiceBus instance for remote communication. And this slowness you describe is typical for a site with a cold cache.

#228139

Sep 19, 2020 9:37

Georg Sandholm

Vote:

Hi Stefan

Thank you for your input! I agree this could be cache related. But I don't understand what would trigger the cache invalidation. We only load some pages during warmup. Nothing that would normally trigger cache invalidation. I will try to add som debugging to look more into the caching.

/Georg

#228191

Sep 21, 2020 7:05

Georg Sandholm

Vote:

After some more troubleshooting we managed to track this down to restart of the production slot in the web app. It happens during Start-EpiDeployment about a minute after the log entry "Preparing target slot for Go Live...". So this happens before Start-EpiDeployment has finished.

We are still trying to find the reason behind this restart. But no luck so far...

#228328

Sep 23, 2020 14:30