Patrick Lam
Jul 19, 2022
(0 votes)

Web Experimentation on World

Half of you might have notice there's another big change to the World homepage, while the other half might not notice much change. Why is that? Over the past few months, some of you might have already notice that you were all part of an experiment that was conducted with our very own Optimizely Web Experimentation. With the results we gathered from the test, we want to adjust the homepage to bring you better content and better engagement with the community.

Purpose of experiment

Since the rebrand of Optimizely, there’s been a redesign to all our digital properties. To ensure that we are engaging with our audience by delivering the relevant content, we wanted to test whether the new design or a design that’s similar to the previous one will have better click through rates and higher page visits. Our hypothesis is that there would not be much difference to our existing users, as they are very familiar with navigating through World. However, for first time new users, the new design will have a slightly better user experience with a better click through rates and page visit count as it gives better direction to where they need to go to find the information they need.


Without having to make changes to the source code and making a deployment, Web Experimentation was a quick way for us to setup and run experiments , especially when we’re testing the presentation layer of a website. Installation was very straight forward, just add a simple script tag which can be generated from the Web Experimentation account by following the 5 simple steps from the documentation:

  1. Decide where to add the snippet
  2. Retrieve the snippet (from the dashboard)
  3. Specify domain
  4. Add snippet
  5. Check if snippet is implemented

* Note: Don’t load the snippet through a tag manager or CMS to keep your website from ‘flashing’ changes

Setup the experiment

Setting up an experiment within Web Experimentation is easy and we already have documentations covering that, so I won’t go into the details here.

However, the setup of a site will be different for everyone, depending on what platform/tech you’re using and how it’s built. For World, we’re using our own Optimizely Content Cloud. I had to make sure to have all the pieces I needed on the site before testing. For example, the experiment we ran on the homepage was to test and see which content and layout will have increase engagement with the visitors. So, all the content (blocks) were added to the page, and then through Web Experimentation, we made two variants where we hid different blocks to create two different layouts on the homepage while keeping both sets of blocks filled with dynamic content from the CMS.

Result of experiment

As we have hypothesized, the results show that there was not much difference in results between the two variations.

Developer focused – Similar to previous design Generic – Broader focus Developer focused Generic broader focus

One metric was to see how many people would visit the Developer Updates page from the homepage. Although less people went to the developer updates page from a developer focused homepage, the experiment result was not significant: 1.91% CTR on the developer focused homepage, and 5.75% CTR on the generic homepage.

Another metric we looked at was the bounce rate. That metric is calculated based on the total bounces (The total number of sessions where visitors exposed to this variation viewed only this page) divided by qualified sessions (The total number of sessions where visitors exposed to this variation started the session on this Page). Again, the numbers did not show any significant differences to prove anything: 32.4% CTR on the developer focused homepage, and 30.6% on the generic homepage.


Most, if not, many of the visitors on World are regulars, and they’re all very familiar with the site and its content. Because of this, most people that visits have a purpose, and they know exactly how and where to fulfil it. Changing the homepage content to this new layout did therefor not significantly impact how people navigate to subsequent pages.

As such, we’ve decided to keep World in a familiar layout to our audiences but adding some minor changes to help share new content that gets updated on World.

Jul 19, 2022


Please login to comment.
Latest blogs
The 1001st Piece in your 1000 Piece Puzzle: .NET Default Interface Functions

I was recently working with a client who wanted a reasonably large subsystem added to Optimizely that would add automated management to their...

Greg J | Nov 28, 2022 | Syndicated blog

Video Demonstration, creating a CMS12 Alloy Sample Site

Hey All Below you will find a quick video demonstration on how to install a local version of Alloy Sample based on CMS12 / .Net 6. As you will see ...

Minesh Shah (Netcel) | Nov 28, 2022

How to create an admin user I Optimizely CMS – with Episerver CLI

In this blog post I’ll show how to create an admin user for Optimizely CMS in a new environment where you don’t have access to the admin interface.

Ove Lartelius | Nov 28, 2022 | Syndicated blog

Optimizely shortcuts in CMS 12 will get a trailing slash appended!

Not all URLs will work when the trailing slash is added, and that could cause problems. Hopefully it will be fixed soon.

Tomas Hensrud Gulla | Nov 26, 2022 | Syndicated blog

One week left of the beta certification period

Today it's one week until the last beta certification day. Do you exam no later than Wednesday the 30th.  Here are the reference and exam guides:...

Karen McDougall | Nov 23, 2022

Unbelievable FREE Heatmapping With Optimizely Web

Within this guide, you will learn how to integrate a free heat mapping tool to Optimizely Web so you can turbocharge your A/B testing capabilities....

Jon Jones | Nov 22, 2022 | Syndicated blog