November Happy Hour will be moved to Thursday December 5th.

DXC deployment times

Vote:
 

Hi all,

I want to share our experience with DXC deployment times in an attempt to confirm if what we are experiencing is "slow" or just "normal".

We are in APAC, and I suspect Azure Resources in the Region might make a big difference. If you reply, I'd love to know what Region you are in.

In our experience, a typical code only DXC Deployment from Integration to Pre-Prod in APAC takes between 40mins and 1.5hrs. The time of day seems to make a difference, out of business hours deployments are often faster than the middle of the day.

Internally, we joke that a typical DXC Deployment from Integration to Pre-Prod is like this;

  • 0-80% = 25% of total duration.
  • 80-81% = 70% of total duration. Warming up the deployment slot.
  • 81-100% = 5% of the total duration.
  • Plus an additional 5-10mins for the completion (slot swap) step

We have raised this with Episerver Support, and are doing some additional logging etc to determine if there is anything unusual occurring.

But more generally, I would be interested to know how the wider Episerver community is finding DXC deployment times.

#205874
Jul 26, 2019 3:28
Vote:
 

Hi Darren

I too find that deploying with DXC is much slower than a regular WebDeploy (like deploying to the Integration environment).

I haven’t seen that slow a deployment before. Usually when I promote from integration to preproduction it takes 15-20 minutes for first step and 5-10 minutes for swapping the deployment slots. And maybe double the time, if including Commerce Manager.

I also found that warming the sites is taking the most time. That is starting the site and rendering the start page is very slow.

So to improve startup times, on some solutions I have done profiling/optimizations of the code that runs on startup. Including custom initialization modules, removing unneeded content types (that is getting synced on startup) as well as optimization of content loading (for instance, loading maybe 40 referenced content items sequentially from database will be slow).

I have found that the databases in DXC (especially the lower plans) are not as fast as on-premise databases. So I try to spend less time talking to them to make both startup and production faster.

#205877
Jul 26, 2019 8:19
Vote:
 

Thanks Stefan. 15-20mins is certainly a lot better than the 40-90mins we've been regularly experiencing. Thankfully including Commerce Manager doesn't double it for us.

I'll check out the startup Content Type optimizations you suggested. But we have a local CI Build Server and its startup time is minimal compared to DXC Deployment, so I suspect its more an issue with Azure deployment slot setup time and the Tier Integration or Preprod is running.

#205884
Jul 26, 2019 9:51
Vote:
 

Yes we also suffer the same issue, we also have it where sometimes the deployments fail after 20-30 minutes warming up a slot and have the whole process start over again. At least the new self service deployments to production are better than having to send an email but it's still quite frustrating how long they take.

#205890
Jul 26, 2019 10:39
Vote:
 

Thanks Scott. So it sounds like most deployments are slow. Maybe our average of 40mins-90mins is just "typical" rather than "unusual" :-)

#205894
Jul 26, 2019 13:29
Vote:
 

I just want to add that I'm not complaining about the DXC deployment time. I'm just keen to understand if this is normal, or if we are experiencing an issue.

#205905
Jul 27, 2019 2:45
Vote:
 

Hi @Darren S

Here is a blogpost created by our engineers describing what's happening during warmup, why it might take some time and some pointers on how to speed it up.

https://world.episerver.com/blogs/anders-wahlqvist/dates/2019/3/web-application-warmup-during-dxc-service-deployments/

Br Emil

#205926
Jul 29, 2019 13:29
Vote:
 

Thanks for the link Emil, it thoroughly explains what's going on during warm-up.

It would be amazing to harvest some data from Deployment Logs etc and create a Power BI a dashboard of DXC deployment times across regions. I can do this locally with my own deployments, but it's value would come with lots of data. 

#205953
Jul 30, 2019 0:09
This topic was created over six months ago and has been resolved. If you have a similar question, please create a new topic and refer to this one.
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.