Blog posts by Polly Walton2026-02-16T15:41:22.0000000Zhttps://world.optimizely.com/blogs/polly-walton/Optimizely WorldWe Cloned Our Best Analyst with AI: How Our Opal Hackathon Grand Prize Winner is Changing Experimentation<p>Every experimentation team knows the feeling. You have a backlog of experiment ideas, but progress is bottlenecked by one critical team member, the Analyst.</p>
<p>Our analyst is the hero of our experimentation programmes. She’s the one who dives into Google Analytics, digs for data and runs the complex calculations for every single test idea. Sample size, baseline metrics, and test duration. It all flows through her. But she’s only one person. The process is slow, repetitive and a classic bottleneck. I can’t count the number of times I’ve jokingly said to her:</p>
<h3><em><strong>“I wish I could clone you!”</strong></em></h3>
<p>That offhand comment sparked an idea. During a brainstorming session with <a href="https://niteco.com/">Niteco</a>'s AI R&D department for the Optimizely Opal Hackathon in New York, we asked ourselves: what would be the biggest-impact solution we could build with AI? I immediately thought: what if we could clone our analyst? What if we could build an AI version of her, an “Experimentation Strategist AI Worker” that could handle the soul-crushing, repetitive work, freeing us up to focus on strategy and insights?</p>
<p>That idea didn’t just get us excited. It won us the <strong>Grand Prize</strong> at the Opal Hackathon in New York.</p>
<p><img src="/link/c2675a2804a148e3ad23ca808e964a5b.aspx" alt="" width="545" height="333" /></p>
<p>I want to take you behind the scenes of our award-winning solution and show you how we’re using Optimizely Opal to build an AI teammate for every experimentation team.</p>
<h2>Meet your new AI teammate: The Experimentation Strategist AI Worker</h2>
<p>At its heart, our solution (which we affectionately call the “Sample Size Genie”) is a specialised AI Worker built natively on Optimizely Opal. It’s not a standalone tool or a fancy chatbot; it’s a fully orchestrated, multi-agent workflow that automates the entire experiment planning process.</p>
<p>It’s designed for anyone on the team, regardless of their technical skill. You simply talk to it in plain English.</p>
<p><strong>You can say:</strong> <em>"Calculate the sample size for a test on our product detail pages, focusing on desktop traffic in the US market."</em></p>
<p><img src="/link/c67115a41ba641a18f74e8cd459597b6.aspx?1771256314210" width="330" /></p>
<p>Instantly, our AI Worker gets to work. It connects to your GA4 property or multiple properties, identifies all the pages that match the “product detail page” pattern, pulls the relevant traffic and baseline conversion metrics, applies the segmentation you asked for, and comes back in seconds with the sample size, estimated test duration and a recommendation on the test’s feasibility.<br /><br /><img src="/link/62af4c0571514f5c83068d28dc26dfb4.aspx?1771256376246" width="330" /></p>
<p>No more spreadsheets. No more waiting for days. Just answers.</p>
<h2>How it works (without getting too technical)</h2>
<p>The magic lies in how we used Opal’s native architecture. We didn’t just write a clever prompt; we built a collaborative team of two specialised AI agents that work together seamlessly.</p>
<h3>1. The Data Acquisition Agent</h3>
<p>This agent is the researcher. Its job is to understand your request, connect to the right data source (such as GA4), retrieve the exact metrics you need and structure the information. If it can’t connect to GA4 or if you’re testing a brand-new feature with no historical data, it’s smart enough to fall back on industry benchmarks (such as Dynamic Yields published benchmarks) or even ask you for manual input.</p>
<h3>2. The Analysis & Recommendation Agent</h3>
<p>This agent is the strategist. It takes the structured data from the first agent and gets to work. It applies the correct statistical formulas, calculates the sample size and test duration and most importantly, assesses the viability of the experiment. If the traffic is too low to yield meaningful results, it won’t just say "no". It will recommend alternatives, like expanding the target audience or adjusting the expected uplift.</p>
<p>These two agents collaborate through <strong>Opal’s Orchestration Layer</strong>, passing data and context back and forth and using a shared Tool Registry to invoke specific functions, such as a calculator or the GA4 Data API. It’s a true AI team, working in perfect sync.</p>
<p><strong>Technical diagram</strong></p>
<p><img src="/link/745cfc869e114961ae3dc4917bfd19f7.aspx" width="330" height="332" /></p>
<p><strong><br />Workflow details</strong></p>
<p><img src="/link/2da433bc0c0843cd891ac6ffc9ce2eba.aspx?1771255480832" width="330" /></p>
<h2>Solving FOUR problems in one go</h2>
<p>What impressed the judges at the hackathon was how this single solution solves four common experimentation headaches:</p>
<ol>
<li>Direct GA4 integration: get sample sizes based on real, live data in seconds.</li>
<li>Limited traffic segments: receive specialist recommendations when testing on niche audiences.</li>
<li>Industry benchmarks: plan tests for new features with confidence, even without historical data.</li>
<li>Manual fallback: keep moving forward even if a data connection fails.</li>
</ol>
<h2>It is a game-changer</h2>
<p>This isn't about replacing analysts. It’s about augmenting them. By automating the 80% of their job that is repetitive calculation, we free them up to focus on the 20% that requires human touch: imagination, deep analysis, strategic thinking and nurturing a culture of experimentation.</p>
<p>We're already building the next evolution of this AI Worker - one that can create many test ideas, generate multiple design variations directly from Figma AI, hold a dialogue with users just like they would with a UX designer about variant designs, and then build the variant code and insert it directly in Optimizely Web Experimentation. Imagine going from nothing to a fully built experimentation roadmap with no analyst, no designer or developer <strong>bottleneck</strong>.</p>
<p>That’s the future of experimentation: a world where Opal empowers our creativity and every team has an AI teammate to help them move faster, test smarter, and deliver better results.</p>
<p>We're planning to give away this and other AI Workers for free, even to companies that aren't <a href="https://niteco.com/">Niteco</a>'s clients. This is so everyone in the Optimizely ecosystem can benefit and grow. As they say, “A rising tide lifts all boats”. By we, I mean the OMVPs at Niteco and by ‘free’ I mean the solution itself will be free, but of course, to use it, you will need to have your own Opal credits. I just want to clarify it. <br /><br />If you are interested in finding out more, reach out to me on LinkedIn. </p>2026-02-16T15:41:22.0000000ZBlog postExperimentation Evolution with AI (Masterclass Recap)<p>If you think you are not using AI in your experimentation program you are probably wrong. Ever asked an AI to rephrase a hypothesis or brainstorm a few ideas? That is a start. But you are just scratching the surface.</p>
<p>The real opportunity is not in automating small tasks. It is in transforming your entire workflow. In a recent Optimizely Academy masterclass I have explained how AI is making experimentation faster smarter and more accessible for everyone.</p>
<p>This article recaps the key insights from my session. It is a practical guide to upgrading your experimentation program with AI to get better results with less effort.</p>
<p><a href="https://academy.optimizely.com/student/page/2820737-masterclass-experimentation-evolution-with-ai">Experimentation Evolution wiht AI - Optimizely Academy Masterclass</a></p>
<p><img src="/link/0587e52d94bb4238834f1d8fc1393f02.aspx" alt="" width="600" height="299" /></p>
<h2>The old way is broken</h2>
<p>Let’s be honest. Traditional experimentation is hard. It is slow expensive and often requires a team of specialists. This creates bottlenecks that kill momentum and cause stakeholders to lose interest. It is a story many of us know all too well.</p>
<p>But it does not have to be this way.</p>
<h2>AI is changing the game</h2>
<p>AI is not a futuristic dream. It is here now and it is changing everything. What used to take days or weeks can now be done in minutes. We are compressing years of effort into weeks.</p>
<p><img src="/link/948eca40157c432baceb8ab71fb20536.aspx" alt="" width="600" height="323" /></p>
<p>AI can help with every step of the process from generating ideas to analyzing results. Here are five areas where AI is making a huge impact.</p>
<h3>1. Idea generation on steroids</h3>
<p>A great experimentation program needs a constant flow of fresh ideas. AI can be your tireless brainstorming partner. Tools like Optimizely Opal and ChatGPT can generate an endless stream of testable hypotheses. The key is to be specific. Give the AI context data and clear objectives.</p>
<h3>2. Calculations without the headache</h3>
<p>Calculating sample sizes and test durations is a massive bottleneck. It is a tedious and error-prone process that can stall your program before it even starts. At Niteco we built an AI agent called the Sample Size Genie to solve this problem. It connects to your GA4 data and provides accurate calculations in seconds. It even offers strategic advice to help you design better experiments. This tool recently won the grand prize at Optimizely’s hackathon in New York.</p>
<p>In the masterclass I showed how I went from a vague idea to a solid experiment plan in under a minute. That is a task that would have taken a specialist analyst an hour to do manually.</p>
<p>I did a demo of how Sample Size Genie works - check it out! All you need is Opal credits. </p>
<h3>3. Design without the delay</h3>
<p>Design is another common bottleneck. But with AI-powered tools like Figma Make you can create professional designs in minutes. Just describe what you want and the AI does the rest. This frees up your designers to focus on more strategic work while you get the high-quality variations you need to test.</p>
<h3>4. Development without the drudgery</h3>
<p>For years code has been the biggest bottleneck of all. But AI is automating the coding process. In the masterclass I have demonstrated how to build a new experiment variant in minutes using an AI tool called v0. This is a game-changer. This functionality is going to be native in Optimizely CMS and Web Experimentation in 2026. The brands that are using AI to eliminate these bottlenecks are running 10 to 20 times more tests with the same resources.</p>
<h3>5. Results analysis on demand</h3>
<p>Analyzing results can be complex. Optimizely’s “Summarize Results” feature is a great starting point. But what happens when a test is not statistically significant? Teams often waste hours manually re-analyzing data in third-party tools.</p>
<p>To solve this we built another AI tool for Opal called the Significance Explorer. It lets you analyze your results using different statistical methods without leaving the platform. This gives you a clearer picture of your experiment’s impact and helps you make smarter decisions.</p>
<p><a href="https://academy.optimizely.com/student/page/2820737-masterclass-experimentation-evolution-with-ai">Watch the Masterclass session recording</a> to get the full insights now.</p>
<h2>The four big benefits</h2>
<p>By embracing AI you will see four huge benefits:</p>
<ul>
<li><strong>Faster time to value</strong>. Get from idea to results in a fraction of the time.</li>
<li><strong>Scalable growth</strong>. Run more experiments and drive more growth without hiring more people.</li>
<li><strong>Higher test velocity</strong>. Remove bottlenecks and dramatically increase your testing rate.</li>
<li><strong>Greater efficiency</strong>. Automate tedious tasks and free up your team to focus on what matters.</li>
</ul>
<h2>Your turn to evolve</h2>
<p>The AI revolution is here. The brands that embrace it will dominate. The choice is yours. You can stick with the old ways or you can evolve with AI and make it your superpower.</p>
<p>Getting started can feel overwhelming. But you do not have to be an expert. Start small. Pick one area of your process and see how AI can help. Then expand from there.</p>
<p>As I said in this masterclass you just have to get to work. Do the math. Solve one problem then the next. The journey of a thousand miles begins with a single step.</p>
<p><img src="/link/fba989c7645240b5aad13e378b30df02.aspx" alt="" width="600" height="339" /></p>
<p>Take that step today or contact <a href="https://niteco.com/">Niteco</a> for help to get you start off on the right foot. </p>2025-11-21T04:45:01.0000000ZBlog post