Prove what your experimentation program is worth with Holdouts

Mon, 30 Mar 2026 16:15:32 GMT

Running experiments is one thing. Knowing whether your experimentation program is moving the business forward? That's the harder question.

Most teams can point to individual test wins. A headline change that lifted click-through. A checkout flow that cut abandonment. But when leadership asks "what is all this experimentation worth?" the answer gets complicated fast.

Today, that question gets a lot easier to answer. Holdouts is now available in Optimizely Feature Experimentation.

What Holdouts do

Holdouts reserves a fixed percentage of your users from all running experiments. That group sees no test variants, no changes, nothing different. They're your global control.

At the end of a measurement period, you compare what happened to the holdout group against everyone else. The difference is the net lift of your entire experimentation program. Not one test. All of them.

It's the cleanest answer to the question every experimentation leader eventually faces: "What would have happened if we hadn't run any experiments at all?"

Why it matters for mature programs

If you're running multiple experiments concurrently, individual test results only tell part of the story. Experiments interact. Variants overlap. And without a clean baseline, you're measuring performance against a constantly shifting target.

Holdouts fix that. It gives you an unbiased comparison point that sits outside your test traffic. The result is a measurement you can take to leadership with confidence.

Not a collection of test summaries. A defensible number that answers the question your CFO is asking.

Built for the programs that need it most

This feature is designed for teams running at scale. If you're running a handful of experiments a year, individual test metrics serve you well. But if your program spans multiple teams, products, and concurrent tests, Holdout Groups gives you the program-level view you've been missing.

And it works alongside your existing setup. No changes to individual experiments. No disruption to running tests. Just an additional layer of measurement that sits above everything you're already doing.

What you can do now

Create a Holdout group in your Flags > Holdouts tab. Set your holdout percentage, select what metrics to measure like Overall Revenue, what Audience, and this holdout will get added to your next experimentation based on the criteria. (I typically recomennd holding back between 1-5% of your traffic). Your next leadership review can include a number that answers the question that matters: what is our experimentation program delivering?

For a full explanation of how holdouts work, see the Holdouts Glossary Page.

Ready to get started? Log in to Optimizely Feature Experimentation.

JavaScript SDK v6: Lightest, Most Efficient SDK Yet

Wed, 11 Jun 2025 21:48:22 GMT

Need a faster site and less frontend bloat? JavaScript SDK v6 is here —and it’s the lightest, smartest SDK we’ve ever released for Optimizely Feature Experimentation.

🚀 What’s New in v6?

Smaller Bundle Size: v6 slashes the bundle size down to as little as 22KB (gzipped) when configured for your needs. This means faster load times and a smoother experience for your users.

Smarter Resource Management: No more manual deactivation in server-side rendering (SSR) environments. JavaScript SDK v6 manages resources more efficiently, simplifying your workflow.

Advanced Decisioning Logic: Get ready for upcoming features like Contextual Bandits, async decision methods, and expanded compatibility with external datastores for sticky bucketing.

⚠️ Before You Upgrade

This is a major version update (v5 → v6) and includes breaking changes.

If you rely on auto-importing the latest SDK version, now’s the time to double-check. You'll likely need to update your imports and integration logic to make the most of the improvements.

Tip: Always pin SDK versions explicitly to avoid surprises when major updates roll out.

🧠 Best Practices and Recommendations

Avoid blindly importing packages using @latest.

Review your dependencies regularly—especially SDKs that affect critical experiences.

Check your SSR logic if migrating from v5—v6 simplifies it, but old patterns may conflict.

✅ Upgrade Today!

Ready to experience the benefits of JavaScript SDK v6? Upgrade now and enjoy a lighter, faster, and more efficient SDK for Optimizely Feature Experimentation.

👉 Check out the JavaScript SDK upgrade guide for everything you need to make the switch.

📚 Need Help?

Our support docs walk you through the upgrade, step by step. Still stuck? Reach out to support or your account team. And don't forgeting to join the community Slack.

Blog posts by Sarah's Blog