Patrick Lam
Jul 19, 2022
(0 votes)

Web Experimentation on World

Half of you might have notice there's another big change to the World homepage, while the other half might not notice much change. Why is that? Over the past few months, some of you might have already notice that you were all part of an experiment that was conducted with our very own Optimizely Web Experimentation. With the results we gathered from the test, we want to adjust the homepage to bring you better content and better engagement with the community.

Purpose of experiment

Since the rebrand of Optimizely, there’s been a redesign to all our digital properties. To ensure that we are engaging with our audience by delivering the relevant content, we wanted to test whether the new design or a design that’s similar to the previous one will have better click through rates and higher page visits. Our hypothesis is that there would not be much difference to our existing users, as they are very familiar with navigating through World. However, for first time new users, the new design will have a slightly better user experience with a better click through rates and page visit count as it gives better direction to where they need to go to find the information they need.


Without having to make changes to the source code and making a deployment, Web Experimentation was a quick way for us to setup and run experiments , especially when we’re testing the presentation layer of a website. Installation was very straight forward, just add a simple script tag which can be generated from the Web Experimentation account by following the 5 simple steps from the documentation:

  1. Decide where to add the snippet
  2. Retrieve the snippet (from the dashboard)
  3. Specify domain
  4. Add snippet
  5. Check if snippet is implemented

* Note: Don’t load the snippet through a tag manager or CMS to keep your website from ‘flashing’ changes

Setup the experiment

Setting up an experiment within Web Experimentation is easy and we already have documentations covering that, so I won’t go into the details here.

However, the setup of a site will be different for everyone, depending on what platform/tech you’re using and how it’s built. For World, we’re using our own Optimizely Content Cloud. I had to make sure to have all the pieces I needed on the site before testing. For example, the experiment we ran on the homepage was to test and see which content and layout will have increase engagement with the visitors. So, all the content (blocks) were added to the page, and then through Web Experimentation, we made two variants where we hid different blocks to create two different layouts on the homepage while keeping both sets of blocks filled with dynamic content from the CMS.

Result of experiment

As we have hypothesized, the results show that there was not much difference in results between the two variations.

Developer focused – Similar to previous design Generic – Broader focus Developer focused Generic broader focus

One metric was to see how many people would visit the Developer Updates page from the homepage. Although less people went to the developer updates page from a developer focused homepage, the experiment result was not significant: 1.91% CTR on the developer focused homepage, and 5.75% CTR on the generic homepage.

Another metric we looked at was the bounce rate. That metric is calculated based on the total bounces (The total number of sessions where visitors exposed to this variation viewed only this page) divided by qualified sessions (The total number of sessions where visitors exposed to this variation started the session on this Page). Again, the numbers did not show any significant differences to prove anything: 32.4% CTR on the developer focused homepage, and 30.6% on the generic homepage.


Most, if not, many of the visitors on World are regulars, and they’re all very familiar with the site and its content. Because of this, most people that visits have a purpose, and they know exactly how and where to fulfil it. Changing the homepage content to this new layout did therefor not significantly impact how people navigate to subsequent pages.

As such, we’ve decided to keep World in a familiar layout to our audiences but adding some minor changes to help share new content that gets updated on World.

Jul 19, 2022


Please login to comment.
Latest blogs
Vulnerability in CMS 12 shell module configuration

Introduction A potential security vulnerability has been identified in Optimizely CMS 12, triggered by a certain shell module configuration. To be...

Magnus Rahl | Sep 28, 2023

AI-Assistant: The 'Change Tone' Shortcut

The AI-Assistant for Optimizely is constantly evolving, adjusting, and transforming to meet your digital needs, providing a cutting-edge advantage...

Luc Gosso (MVP) | Sep 27, 2023 | Syndicated blog

How to fix scheduled job 'Remove Abandoned BLOBs' if it keeps failing

Optimizely inlcudes a job named 'Remove Abandoned BLOBs'. This post will help you fix it if it's no longer working.

Henning Sjørbotten | Sep 26, 2023 | Syndicated blog

Optimizely Web Experimentation Metrics

Within the domain of Optimizely Web Experimentation Metrics, the emphasis is on objective key performance indicators (KPIs) selected to assess an...

Matthew Dunn | Sep 23, 2023 | Syndicated blog

Optimizely For you Intranet

Having been at Optimizely and in the CMS industry for nearly 16 years I have seen clients’ intranet requirements go from a simple site just to hous...

Robert Folan | Sep 22, 2023

Vulnerability in EPiServer.GoogleAnalytics v3 and v4

Introduction A potential security vulnerability was detected for Optimizely Google Analytics addon (including EPiServer.GoogleAnalytics and...

Bien Nguyen | Sep 20, 2023