Web Experimentation on World

Half of you might have notice there's another big change to the World homepage, while the other half might not notice much change. Why is that? Over the past few months, some of you might have already notice that you were all part of an experiment that was conducted with our very own Optimizely Web Experimentation. With the results we gathered from the test, we want to adjust the homepage to bring you better content and better engagement with the community.

Purpose of experiment

Since the rebrand of Optimizely, there’s been a redesign to all our digital properties. To ensure that we are engaging with our audience by delivering the relevant content, we wanted to test whether the new design or a design that’s similar to the previous one will have better click through rates and higher page visits. Our hypothesis is that there would not be much difference to our existing users, as they are very familiar with navigating through World. However, for first time new users, the new design will have a slightly better user experience with a better click through rates and page visit count as it gives better direction to where they need to go to find the information they need.

Installation

Without having to make changes to the source code and making a deployment, Web Experimentation was a quick way for us to setup and run experiments , especially when we’re testing the presentation layer of a website. Installation was very straight forward, just add a simple script tag which can be generated from the Web Experimentation account by following the 5 simple steps from the documentation:

1. Decide where to add the snippet
2. Retrieve the snippet (from the dashboard)
3. Specify domain
4. Add snippet
5. Check if snippet is implemented

* Note: Don’t load the snippet through a tag manager or CMS to keep your website from ‘flashing’ changes

Setup the experiment

Setting up an experiment within Web Experimentation is easy and we already have documentations covering that, so I won’t go into the details here.

However, the setup of a site will be different for everyone, depending on what platform/tech you’re using and how it’s built. For World, we’re using our own Optimizely Content Cloud. I had to make sure to have all the pieces I needed on the site before testing. For example, the experiment we ran on the homepage was to test and see which content and layout will have increase engagement with the visitors. So, all the content (blocks) were added to the page, and then through Web Experimentation, we made two variants where we hid different blocks to create two different layouts on the homepage while keeping both sets of blocks filled with dynamic content from the CMS.

Result of experiment

As we have hypothesized, the results show that there was not much difference in results between the two variations.

Developer focused – Similar to previous design Generic – Broader focus

One metric was to see how many people would visit the Developer Updates page from the homepage. Although less people went to the developer updates page from a developer focused homepage, the experiment result was not significant: 1.91% CTR on the developer focused homepage, and 5.75% CTR on the generic homepage.

Another metric we looked at was the bounce rate. That metric is calculated based on the total bounces (The total number of sessions where visitors exposed to this variation viewed only this page) divided by qualified sessions (The total number of sessions where visitors exposed to this variation started the session on this Page). Again, the numbers did not show any significant differences to prove anything: 32.4% CTR on the developer focused homepage, and 30.6% on the generic homepage.

Conclusion

Most, if not, many of the visitors on World are regulars, and they’re all very familiar with the site and its content. Because of this, most people that visits have a purpose, and they know exactly how and where to fulfil it. Changing the homepage content to this new layout did therefor not significantly impact how people navigate to subsequent pages.

As such, we’ve decided to keep World in a familiar layout to our audiences but adding some minor changes to help share new content that gets updated on World.