DXP deployment - warmup always times out

November Happy Hour will be moved to Thursday December 5th.

AI OnAI Off

Home / Forums / Developers (Main Products) Forum / CMS 7.5 - 11 /

DXP deployment - warmup always times out

Jerzy Czopek

Vote:

We have a website hosted on DXP platform. None of the website is available anonymously - every vistor has to be logged.

Therfore the deployment takes ages, because it waits for timeout, and this things get logged:

Timed out waiting for all instances for webapp **************inte and slot "slot" to become ready!

2020-09-10 09:43:23 	 ::PROGRESS:: 	 PercentComplete=80
2020-09-10 09:43:23 	 Information 	 Starting to warm up the targets slots...
2020-09-10 09:43:50 	 Information 	 Preparing target slot for Go Live (**************/slot) (warming up the slot)
2020-09-10 09:55:11 	 Information 	 Starting any continuous web job(s) in the slot if they exist (**************/slot)
2020-09-10 09:55:13 	 Information 	 Sending webrequests to validate the slot (**************/slot)...
2020-09-10 09:55:47 	 Warning 	 Site with URL http://**************-slot.dxcloud.episerver.net/ responded with error code: 403 (Forbidden)
2020-09-10 09:55:47 	 ::DATA:: 	 SiteName=**************;URL=http://**************-slot.dxcloud.episerver.net/;ManualValidationLink=http://integration.XYZ.com/?x-ms-routing-name=slot;StatusCode=403;StatusDescription=Forbidden
2020-09-10 09:56:39 	 Information 	 Warmup finished for all web apps!
2020-09-10 09:56:39 	 Information 	 Re-enabling autoscaling for the web apps in resource group **************
2020-09-10 09:56:45 	 ::PROGRESS:: 	 PercentComplete=90

So what I did, I created a simple API controller and action, which allows anonymous access, that loads start page through ContentLoader and direct children and returns list of loaded pages with status code OK 200.

Warmup controller looks like this:

    [AllowAnonymous]
    public class WarmupController : ApiController
    {
        
        [HttpGet]
        [Route("api/warmup")]
        public IHttpActionResult Index()
        {
            // populate cache by retrieving start page and all direct child pages
            var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
            
            var pageSegments = new List<string>();
            
            var startPage = contentLoader.Get<PageData>(ContentReference.StartPage);
            if (startPage == null) return Ok<IEnumerable<string>>(pageSegments);
            pageSegments.Add(startPage.URLSegment);

            // load child pages
            var childPages = contentLoader
                .GetChildren<PageData>(startPage.PageLink).Where(x => x != null)
                .Select(x => x.URLSegment);
            pageSegments.AddRange(childPages);

            return Ok<IEnumerable<string>>(pageSegments);
        }
    }

I have added following initialization entry into transform file:

 <!-- Application initialization -->
    <applicationInitialization xdt:Transform="Remove" />
    <applicationInitialization xdt:Transform="InsertIfMissing">
      <add initializationPage="/api/warmup" hostName="integration.XYZ.com" xdt:Transform="InsertIfMissing" />
    </applicationInitialization>

When accessing manually, https://integration.XYZ.com/api/warmup, I can confirm it's allowing anonymous access and it returns what it's supposed to. However from the logs, seems like it tries to hit the root of the page any ways, and it will always Forbidden because anonymous access is not allowed there due to bussiness requirements.

How to solve this issue? I bascially would like to shorten the warmup so that it doesn't take more than a half of the whole deployment time because of the forbidden URL. Ideally I would like the warmup to hit the endpoint I created for that reason.

#227694

Sep 10, 2020 10:24

Stefan Holm Olsen

Vote:

Hi Jerzy

Can you confirm that DXP is even calling your new API endpoint? You didn't mention it, and by default application initialization will hit the start page first (which gives the 403).

You could check that the web.config file has been correctly transformed with your additions.

#227713

Sep 10, 2020 18:07

Jerzy Czopek

Vote:

Seems like it's not hitting that URL and it's indeed hitting start page, which deliberately is set to only allow authenticated traffic.

So now the question is: How could I authorize warmup requests to the page? OWIN middleware?

#228106

Sep 18, 2020 9:53

Stefan Holm Olsen

Vote:

Then it does not sound like an authentication issue. More like a wrong configuration of the applicationInitialization configuration.

You could try connecting to the integration site with FTP and verify that the config transform you made is actually applied.

#228107

Sep 18, 2020 10:00

Jerzy Czopek

Vote:

After further digging, I found out that it is calling my warmup URL (which takes about 4 seconds to complete), but yet it's still reporting timeout with the page URL which to me looks like it hits the start page anyway - even though it is not listed in applicationInitialization .

Application initialization looks like this, so it's properly transformed (from what I can tell):

<applicationInitialization>
      <add initializationPage="/api/warmup" hostName="integration.XXX.com"/>
    </applicationInitialization>

Any ideas?

#228110

Sep 18, 2020 12:33

Anders Wahlqvist

Vote:

Hi @Jerzy,

This might be tricky for you to solve, the deployment engine will currently always try to hit the start page of the domain when it tries to validate warmup. Warmup itself however should work as expected though (the deployment engine just tries to validate when/if this process has finished).

So to clarify: Application Initilization will go through the links you've added to the config (this is the actual warmup that also happens during scale outs). The deployment engine will ping the start page (it should follow redirects within the domain though) to try to monitor this (and check the header the app init process adds when it's running, which sadly is only returned if the response is 200 OK). If we can't validate that warmup has finished, we will ping the site until we hit our timeout value which seems to be the case here, it's not optimal but it's the safest way we can do this when we can't validate the warmup progress.

I don't have any good suggestions for how to workaround this problem currently (except for having a login page that responds with 200 OK, then this should be resolved). But we do have some planned work in this area ahead of us, so even though I can't make any promises on when/if this particular scenario will be fixed, I do have some hope of addressing this and will try to keep this scenario in mind when we do.

Small sidenote though is that even the fastest sites need a couple of minutes to warmup, initiate local cache and other internal processes in Azure etc., so we're really "just" talking about a few minutes of extra deployment time because of this, but I do of course get why you'd like this to be as quick as possible, and we do want to make this as fast as we possibly can as well.

Just saying that the entire time the deployment stays in this warmup validation phase isn't waste if that makes sense :-)

#228200

Sep 21, 2020 10:09

MMendes

Vote:

Also it is very common for the lower environments to have an IP white list and that causes the same type of problems.
Any chance we can specify in the EPI Cloud API which page the Warm Up stage should reach ?

#251006

Mar 19, 2021 19:24

MMendes

Vote:

I think that alone would be enough to solve the problem.
If I am not wrong the IIS AppInitialization runs from the localhost, so adding the localhost to the withelist of IP;s would effectivelly eliminate all 403's.

Also From our side we could create a page to be reached by the warm up stage containing all the links we really care, not only the ones in the home page.

Sounds like a Win Win situation.

#251007

Edited, Mar 19, 2021 19:24

Anders Wahlqvist

Vote:

Hi @MMendes,
We do have some improvements to this process planned for the future where we'll try to figure out what pages are most commonly hit by traffic and warmup those primarily instead, and also filter hostnames based on the site configuration in the CMS. But as far as customization goes today, it's pretty much "all or nothing". You can add your own App Init section where you have full controll over what links will be added, or we'll use the automatic method we have by default. This does seem to work well for the majority of all implementations.

It's good to note that the App init process in the IIS as you say runs locally and even bypasses a few modules, but always good to exclude localhost for blocks anyway I guess. Our validation process during deployments run externally though and that infrastructure uses dynamically assigned IPs so that's a bit harder to solve, and it's that validation process that got a 403 response as mentioned in this thread (warmup itself might've worked better).

Hope that makes sense, apologies if I missunderstood what you meant!

#251093

Mar 22, 2021 10:48

Jørgen Helgheim

Vote:

I have the same problem for my customer. Does DXP offer any custom request headers in the warmup requests? Or does it have a custom user agent string? Then I could use this to allow access to the prep-site. ex

<rewrite>
<rules>
 <rule>
  <condition>
   <add input="{HTTP_USER_AGENT}" pattern="Initialization" negate="true"/>

#267316

Nov 24, 2021 14:43

Anders Wahlqvist

Vote:

@Jørgen Helgheim: All requests that we send to the site during deployments are using the user agent string: "Episerver DXC Automation Engine"

So you could try that! Please be aware that the actual warmup requests issued by Azure will use a different user agent string though, but the validation requests that we send will use that one.

#267537

Nov 29, 2021 11:56

Please login to post a reply