Accelerating Optimizely CMS and Commerce upgrades with agentic AI (Part 1 of 2)

Image: Niteco agentic Upgrade Machine (CMS 11 and Commerce 13 → CMS 12 and Commerce 14) 

TL;DR

We created the Upgrade Machine, an AI system with a main orchestrator, specialized subagents, and a growing skills library. It speeds up the most repetitive part of an Optimizely upgrade by producing a PR-ready codebase that builds in Release mode and starts the CMS backend cleanly on the target platform.

After that, a Niteco delivery team steps in to stabilize integrations, verify behavior, and move the project to production. This usually takes 2 to 6 weeks, depending on the project's size.

1. Why Optimizely upgrades stall (and why timing matters now)

Upgrading Optimizely is more than just updating a package. It affects runtime assumptions, custom code, integrations, routing, dependency injection, configuration, content models, and readiness for cutover. Moving from CMS 11 and Commerce 13 to CMS 12 and Commerce 14 is a big change because it is part of a larger platform update. Stable patterns may suddenly need major changes.

There is also a lifecycle reality. Optimizely’s policy is that all new functionality is built only for the latest major CMS version (currently CMS 13). Optimizely actively monitors and fixes severe bugs and security issues only for the current major version and one prior major version  (currently CMS 13 and CMS 12). Older versions such as CMS 11 do not receive ongoing bug monitoring, standard bug fixes, security monitoring, or regular patch releases, except in exceptional cases involving severe security vulnerabilities raised by customers/partners. Practically, delaying the upgrade increases platform and security risk because you’re operating outside the normal maintenance and patch cadence.

From our experience, the biggest delays come from discovery gaps like undocumented integrations, old helper code, and tricky configuration wiring. These issues slow teams down with repeated compile-fix cycles and unexpected problems late in the process. Our goal with the Upgrade Machine is simple: take the repetitive refactoring out of the main workflow and give engineers a clean, buildable starting point.

2. What automation solves, and where it stops

Rule-based upgrade tooling is genuinely useful for deterministic transformations. Retargeting frameworks, aligning package references, updating project files, and applying known API substitutions should be automated. Most upgrades should start there, and ours do.

Problems arise when upgrades involve complex work, such as changes across files, unclear intent, or cases where several fixes might work but only one fits the platform's needs. Rule engines can't understand intent- they either make a safe change or escalate the issue. These escalations often pile up, creating a manual backlog that can last for weeks. This is where agentic AI makes a real difference.

3. The agentic approach: what we built (part 1 focus)

The Upgrade Machine is a coordinated system of AI coding agents working within a managed workflow.

·    A main agent orchestrates the run: it sequences phases, enforces quality gates, and tracks progress in a persistent log.

·    Specialist subagents handle narrow domains: code fixes, reviewer checks, unit-test scaffolding, and skill enrichment.

·    A shared skills library encodes typical upgrade patterns derived from real delivery work. Each skill has clear triggers, a defined transformation approach, and validation rules.

Two design choices matter most.

First, agents have autonomy but with clear limits. They can work on their own, but only within a process that includes checkpoints, reviewer checks, and escalation points. If the system is unsure, it stops and asks a developer to decide instead of making a guess.

Second, the system gets better over time. Each project adds new patterns to the skills library. The system improves by capturing what works, testing it in different situations, and turning it into safe automation.

For this post (part 1), we define 'done' as follows: the solution builds in Release mode on the CMS 12 and Commerce 14 targets, and the CMS backend starts without issues.

4. Upgrade Machine, an agent harness architecture

Agent harness (execution layer)

Behind the visible agents and tools is the agent harness. This is not just a model - it is the part that makes agentic work safe and repeatable in real engineering settings. It provides:

·    Run isolation and reproducibility: creates an isolated workspace and enforces a checkpointed, branch-based flow so changes are easy to audit, pause, and resume.

·    State and context management: packages the minimum required context for each subagent, tracks progress state, and prevents uncontrolled context drift across long runs.

·    Tool governance: controls which tools an agent can use, in what sequence, and under what policy constraints (build, scan, test, report) so automation stays deterministic.

·    Quality gates and escalation: enforces reviewer gates and uncertainty thresholds, escalating to a human decision when intent is ambiguous or risk of behavior change is high.

·    Observability and artifacts: collects structured logs, code differences, and run outputs like reports, checklists, and lists of remaining decisions. This helps delivery teams stabilize and deploy with confidence.

Skills library

This acts as the team's memory. It holds proven patterns in areas like migration, CMS and Commerce, dependency injection, configuration, and integration clients. Patterns go through a process: they are first observed, then validated, and finally marked as safe for automation.

Specialist subagents

These are narrow, purpose-built agents invoked by the orchestrator:

·    Code-fix agent: applies migration fixes guided by the skills library

·    Reviewer agent: validates output against the skills library and rejects fixes that compile but violate migration intent

·    Unit-test agent: scaffolds and runs tests where feasible

·    Skill-enrichment agent: captures new patterns discovered during the run and feeds them back into the skills library

Toolchain integrations

Agents do not work in isolation. They use real tools like dotnet build, dependency scanners, test runners, and report generators. All outputs are logged and included in the run report.

Main agent

The orchestrator plans and drives the run, assigns work, tracks state and progress logs, enforces guardrails, and compiles the final report for developers and stakeholders.

Build-fix loop

This is the core of the process. Build results are checked against a migration debt index, fixes are made, reviewer checks are done, and checkpoint commits are created. This loop continues until the debt index is cleared. Only then does the workflow move to Release build verification.

5. How a run works end-to-end (CMS 11 and Commerce 13 → CMS 12 and Commerce 14)

A run moves through phases with explicit exit conditions and artifacts.

Phase 1: pre-flight and isolation

The orchestrator assesses solution structure, dependencies, and likely breaking zones. Work runs in an isolated branch. Baselines remain untouched. Every phase produces checkpoints, so a run is auditable and resumable.

Phase 2: guided migration

The machine upgrades legacy patterns to modern ones and updates dependencies for the target platform. In this phase, getting the target platform right is more important than just making the code compile.

Phase 3: continuous validation loop (the heartbeat)

This is where agentic AI proves its worth.

·    The orchestrator builds and classifies issues against migration debt patterns.

·    Code-fix is dispatched in controlled batches.

·    A reviewer gate validates that fixes align with accepted patterns, not quick shims.

·    Checkpoints are committed only after reviewer acceptance.

·    If progress stalls or intent becomes ambiguous, the system escalates to a developer with options and rationale.

Phase 4: Release build verification and CMS boot validation

A Debug build alone is not enough. Building in Release mode and checking CMS startup helps catch wiring issues early. The process repeats until the Release build is clean and the CMS backend starts without problems.

Phase 5: Output artifacts

At completion, the machine produces:

·    PR-ready branch with checkpoint commits

·    Structured change log grouped by pattern families

·    Build logs and run summary

·    Remaining decisions list (items intentionally escalated to humans)

·    Stabilization checklist tailored to the solution footprint

·    Skill enrichment notes for future runs

6. Skills library: deep enough to trust, not detailed enough to copy

The skills library is the most valuable part of the system and is why results get better over time. Each skill is a documented pattern with:

·    Trigger: how the system detects the pattern (error signature, code shape, dependency signal)

·    Transformation: how code is changed safely

·    Validation: how correctness is verified beyond “it compiles”

Skills do not start out as automated. Patterns are collected from real projects, tested in different codebases, and only then marked as safe for automation. This approach avoids guesswork and helps prevent hidden problems.

Representative categories include:

·    Project and dependency modernization patterns

·    DI and configuration patterns aligned to modern platform expectations

·    Routing and runtime initialization patterns

·    Integration client modernization patterns (safe client setup, resilience, async correctness)

·    Commerce-related upgrade patterns where the upgrade path diverges

·    Safe cleanup patterns that require explicit human sign-off

7. Human-in-the-loop and guardrails

Agentic systems need to be able to pause. Our system is built to escalate issues when needed.

When the orchestrator reaches a point of uncertainty, it stops and asks for input. It suggests options and tradeoffs, and only moves forward after a decision is made.

Guardrails ensure changes remain safe and auditable:

·    PR-style diff flow

·    Checkpoint commits at every phase

·    Full action logs

·    Reproducible runs

·    Mandatory human review before merge

·    No deployment actions by the machine

For engineering leaders, the risk profile is familiar. It is a disciplined, branch-based workflow, just much faster.

8. Definition of done and the handover model (part 1)

We are deliberate about boundaries.

Done for Part 1 (fully autonomous agentic workflow) means:

·    The solution builds successfully in Release mode on the CMS 12 and Commerce 14 target

·    The CMS backend starts cleanly

·    Artifacts are produced: change log, build logs, remaining decisions list, stabilization checklist

What the delivery team does next (typically 2 to 6 weeks)

·    Stabilize regressions surfaced by functional and exploratory testing

·    Validate integrations, scheduled jobs, and data flows

·    Address performance and security hardening as required

·    Prepare deployment, rehearse cutover, and ship to production

This division keeps the model reliable. The machine speeds up the foundational engineering work, while the delivery team is responsible for making sure the solution is ready for production.