Blog posts by Holly Quilter

The Experimentation Process

Thu, 16 May 2024 01:13:53 GMT

This blog is part of the series - Unlocking the Power of Experimentation: A Marketer's Insight.

Welcome back, to another insightful journey into the realm of experimentation. In this edition of our series, we're rolling up our sleeves and digging into the experimentation process. Let's unpack how we tackle experimentation to deliver optimization at scale and generate actual ROI value.

The Role of Data in Experimentation

Before we delve into the specific steps, it's crucial to acknowledge the backbone of the entire process: DATA. Data fuels every stage of our experimentation journey. It informs our hypotheses, guides our strategies, and provides the metrics we need to evaluate our results. By leveraging data effectively, we ensure our experimentation efforts are grounded in reality and focused on delivering tangible improvements. With data as our compass, we can navigate the complexities of user behaviour and continually refine our approach to achieve optimal outcomes.

Step 1: Goal Setting

Every successful experimentation journey begins with clear goal setting. We kick off by defining our objectives: What are we trying to achieve? What impact do we want to have on our users? Identifying friction points in the user journey helps us create a goal tree — an evolving roadmap that aligns experimentation with business objectives. This step is crucial for measuring and analyzing performance against key KPIs and metrics, setting the stage for success.

Step 2: Hypothesis

With our goals in sight, we dive into hypothesis generation. Drawing on data insights, we formulate predictions about the behaviour of measurable outcomes. A strong hypothesis consists of three parts: defining the problem, proposing a solution, and predicting the result. By aligning our experiments with well-crafted hypotheses, we set the stage for meaningful insights and impactful optimizations.

Step 3: Experiments and Tests

Armed with hypotheses, we move into the experimentation phase. Here, we launch experiments with appropriate controls, targeting, and tracking to validate our hypotheses and uncover statistically significant outcomes. From classic A/B/n testing to user testing, phased rollouts, and personalization, we explore diverse testing methodologies to refine the digital experience and drive desired outcomes.

Step 4: Analysis

Once experiments are underway, it's time for analysis. We delve into the data, collating results, conducting thorough analysis, and generating insights into the performance patterns we observe. Statistical significance guides our decision-making, ensuring that observed results are attributable to the changes we've made, not random chance. These insights serve as the foundation for informed decision-making and pave the way for further optimization.

Step 5: Continuous Optimization

So we do not stop there! We embrace an always-on optimization mindset through iterative cycles of experimentation. This continuous loop — from data analysis to experimentation technology and back again — fuels our pursuit of ongoing improvement. With each iteration, we refine the digital experience based on proven data-driven insights, driving sustainable growth and enhancing user satisfaction.

In conclusion, the experimentation process is a journey fuelled by curiosity, data, and a commitment to continuous improvement. By mastering the art of goal setting, hypothesis crafting, rigorous testing, insightful analysis, and always-on optimization, marketers can unlock the full potential of experimentation to drive meaningful results and deliver exceptional user experiences. Stay tuned for more insights, strategies, and experimentation magic in our ongoing series. Until next time, happy experimenting! 🚀✨

The distinctions between Optimizely Web Experimentation and Optimizely Feature Experimentation

Tue, 09 Apr 2024 23:08:26 GMT

This blog is part of the series - Unlocking the Power of Experimentation: A Marketer's Insight.

Let’s dive into Optimizely's powerful experimentation tools and unravel the nuances between the two key offerings: Optimizely Web Experimentation and Optimizely Feature Experimentation. Both Web Experimentation and Feature Experimentation are integral parts of the Optimizely platform, designed to evangelize data-driven decision-making and empower organizations in their pursuit of optimizing user experiences.

Optimizely Web Experimentation: Elevating User Experience

Optimizely Web Experimentation serves as a client-side experimentation tool, ideally suited for marketing or growth teams aiming to optimize websites. The accessibility of this tool is a game-changer, as it caters to users without strong technical skills. The WYSIWYG visual editor enables teams to bring their experimentation ideas to life—from hypothesis to results—without the need for a single line of code. It is typically used to optimize conversion rates, boost engagement, and enhance user journeys. Web Experimentation encompasses diverse tactics from A/B testing page layouts and design elements, to experimenting with different copy, images, and user flows. The goal is to uncover the most effective combination that resonates with your audience and drives desired outcomes.

Optimizely Feature Experimentation: Empowering Product Development

Optimizely Feature Experimentation, in contrast, is an omni-channel experimentation tool that caters to any device with an internet connection. It is ideal for product development teams seeking to rollout or test specific functionalities, features, or even entire product offerings without incurring any performance degradation. Users of Feature Experimentation prioritize metrics like engagement, retention, usage, and lifetime value, contributing to a holistic product optimization strategy rather than focusing solely on conversion rates. It is a code-centric tool that provides configurable architecture, known as feature flags, to support development releases across digital platforms and technologies.

Choosing the Right Approach for Your Goals

In conclusion, the choice between Web Experimentation and Feature Experimentation depends on your specific objectives. If your aim is to enhance the visual and navigational aspects of your website as part of your conversion rate optimization strategy, Web Experimentation is the ideal path. On the other hand, if you are seeking to innovate and refine product functionalities, Feature Experimentation offers a robust framework for continuous improvement.

By harnessing the strengths of both tools, organizations can create a comprehensive optimization strategy that addresses both user experience and product development. As experimentation analysts, let's leverage the versatility of Optimizely to unlock insights, drive innovation, and deliver digital experiences that resonate with our audiences. Happy experimenting!

Unlocking the Power of Experimentation: A Marketer's Insight

Mon, 01 Apr 2024 16:38:28 GMT

Ever scrolled through the Optimizely Community blogs and felt like you’ve stumbled into a developer’s journal? Yep, well I’ve been there too. But guess what? I think it's time to shine a spotlight on the perspective of marketers.

Welcome to a series that delves into Experimentation with Optimizely from the lens of a marketing professional navigating the digital world of optimization – let the exploration begin! 🚀✨

Why a Series for Marketers?

This short blog series aims to empower the analysts, the strategists, the curious, and the open-minded with the knowledge and tools needed to leverage experimentation effectively in their marketing endeavours. Whether you're embarking on setting up an experimentation program for the first time, implementing Optimizely’s Web or Feature Experimentation, on a mission to foster a culture of innovation or just want to learn more - this series is tailored for you.

Demystifying Experimentation

Taking it back to basics, experimentation is defined as testing ideas scientifically, just like how we did all those years ago back in school in our science labs. You attempt to answer a question by establishing a hypothesis, test that hypothesis through experimentation and analyze the results. In the digital world, we are doing exactly the same thing, but instead of bunsen burners and chemicals, we are testing and developing user experiences.

Digital Experimentation encompasses a variety of tactics designed to enhance the user experience through data-driven hypotheses. There are three types of experimentation:

Experiment: To discover without fear of failure or expectation of an outcome.
Test: To validate our assumptions and measure impact, predicting expected winners or outcomes.
Personalize: To tailor the user experience based on defined criteria.

Insights gained from experimentation enable organizations to make decisions backed by data. Optimization is the goal; experimentation is the mindset for how we get there. A great explanation of the experimentation mindset can be found in Kevin’s article here. The article emphasizes that experimentation is not limited to scientific labs and there is a VERY simple framework inspired by Sesame Street that helps lay the foundation:

I wonder…
What if…
Let’s try!

Cultivating Curiosity

You will often find the obstacles of experimentation are not tools or technologies but rather the behaviours and opinions of this who do not understand its value. I mean, there must be a reason why major players like Netflix and Amazon are dedicated to experimentation right? It is no longer just a strategy; it is a competitive advantage. As stated by the Harvard Business Review, "To successfully innovate, companies need to make experimentation an integral part of everyday life. That means creating an environment where employees’ curiosity is nurtured, data trumps opinion” and experimentation is embraced. Hopefully, this series will help you accomplish that!

So, join me on a journey to unlock the power of Experimentation with Optimizely, enabling you to set your clients up for success. Welcome aboard! 👋