How to load test your sites at DXP

What is the best load-testing strategy - to invest in a realistic test or take a quick-and-dirty approach? Exactly where to cut away complexity in your load testing depends on your situation, but you may want to use a simplistic version of user behavior when designing your load test.

The simulated user in your test might be more static in its behavior than a real user, perhaps just accessing a set sequence of pages or content with no randomization. Or your simulated users access a randomly selected page on your site, choosing to ignore the fact that some pages are much more heavily visited than others.

Why should you load test your sites?

At Optimizely, load tests are something that represent an accurate customer journey on your website. Simulated user volume should align with historical or expected site traffic including historical ramp up and down times such as visiting the home page, navigating to another page, searching, adding items to carts, and so on.

When should you do your first load test?

Load testing your solution is also something you do before you go live with Optimizely DXP for the first time. You, the client, or your partner, or a third party of your choice should complete load performance testing at least two weeks before Go Live. Ideally earlier than two weeks, as soon as dependent apps are integrated it is a good point too, as it will pick up any bottlenecks in the overall solution long before you get near the go live date. Two weeks is the time to either optimize and resolve identified problem on the platform or the application from load testing results, or to make necessary decisions about whether the planned Go Live date must be postponed. 

When should you do your second load test?

Well, after the first one. The important thing is that you don’t stop at test #1. We see that customers that continuously test and iterate their code have more healthy applications which means better user experience, web performance, SEO and security. For most customers and partners is Continuous Integration/Deployment (CI/CD) the path forward to overcome concern around deployments. Load testing is often times just another step to the process.

In our DXP platform you’ve three environments, the pre-production environment is ideal for load tests without impacting your production website. We scale up your pre-production environment to mimic your production environment setup for testing purposes, to later scale it back when the testing is done.

What scenarios should you test?

A test can measure the following scenarios, depending on your expected traffic to your website:

• A gradual increase of traffic that tests auto-scaling.
• A burst of traffic that tests a planned campaign where you know that you will receive increased traffic during a period of time. Optimizely Support pre-scales the environments so it does not have to fully rely on the auto-scaling feature.

Note: Testing unusual loads – for example, 1500 virtual users visiting 15 different pages, refreshing all of them every second for 10 minutes – should be reserved for testing a site against an attack; not typical load scenarios.

Test end-to-end simulations (not just test one part of a site), especially on a commerce site; user checkout needs to be withing scope. For example, it is not an accurate simulation to test only triggering APIs on the site. All dependencies should be within the scope of the test, such as Search & Navigation, external dependencies, and so on..

Consider the following test criteria

• Perform a discovery or inventory of what is in and out of scope for your test.
• Iterative testing improves results by looking for bottlenecks either in the infrastructure or application code.
• A real user scenario should have sleep timers between each action.
• Virtual users should not do the same action at the same second.
• Each virtual user should have its own set of cookies. If virtual users share the same cookies, they will hit the same app service instance which guarantees a crash.
• Use Azure Application Insights to examine the telemetry of requests and traffic from the load test.
• Make sure to warm up the CDN cache to not get skewed results from uncached content (that will be cached in reality)

How should you act on the results?

You can realize performance improvements by maximizing your use of CDN caching and specifying the number of results returned from Search & Navigation or how often a search can be used. If the first load testing did not “meet the bar,” then iterate with improvements and do it again. 

When working on improving performance in complex applications, there is rarely one big change that you can make that will make everything immediately faster or scale better. Many small incremental changes built up over time are usually required to achieve your goals, make use of Continuous Integration/Deployment (CI/CD) to achieve this way of working. Those changes can have hidden trade-offs and it’s not always clear how much a given change will improve things. Experimentation can tell you with certainty that you are making things better, also when doing backend improvements.

Consider automatic scaling of the DXP platform as a factor, because short high-intensity load tests will not test the DXP platform’s ability to scale the application under a growing load. The DXP platform scales out instances based on resource use over an aggregate period. DXP-specific considerations are scaling, caching, CDN, warm-up/code initialization when a new instance is spun up, and dependencies (such as Search & Navigation). 

Another important step is to involve our support team for scaling down auto-scaled instances in case they want to re-run a test immediately instead of having to wait for instances to scale-in to minimum value (can take hours).

Do not hesitate to engage the Optimizely teams for questions around load testing.  

More reading on the subject.

The following blog posts describe other tools to enhance your understanding and preparedness for scaling, using feature flags and rollout tests.

• Feature Flags: 3 Use Cases for your Development Cycle - by Optimizely
• Using Experimentation to Measure and Validate Backend Performance Improvements - by Optimizely
• The easiest way to create realistic load testing scrips - by K6, a load testing tool we use ourselves at Optimizely
• Difference between Load Testing and Stress Testing - by GeeksforGeeks

Feel free to share your experiences and best practices in the comments section below.

Happy testing!

John Håkansson

Global VP Product Management for CMS & Cloud Platform at Optimizely