We are introducing EPiServer A/B Testing module, but there are so many issues with it I almost don't know where to start.
We publish content at page level - rather than block level - but when we run an A/B test, we have to publish new blocks. We cannot publish an existing, but edited, block because the test does not show the block. Instead a new block has to be created from scratch for the test to run - even if we just want to change one word. RIDICULOUS!
Also, we cannot run any form of workflow/approval process on the challenger page before it goes live. Big issue when you are a financial services company like us, and we need an audit trail for all of our webpages...
Finally the module does not work with fetching pages/content which is how we have set up our CMS.
Any solutions to the above would be greatly appreciated.
The latest Episerver A\B test package does allow for block level testing in the same way as page level testing. What process are you going through that you are able to test pages but not blocks?
The interaction between A\B testing and workflow approvals is a sequential approach. Use the workflow approval process on content edits then on the last workflow step the content should be in a ready to publish state if approved, then use the A\B test option in the publish menu to A\B test the approved draft version. This is because, as you mention, the A\B test allows the draft version to go live on the site for a portion of your visitors, so any approval has to occur before running a test. This is also the reason why we check to see if a user has publish rights to the content in order to setup or start an A\B test as a user trusted with publish rights should know when the draft content is ready to be viewed by site visitors.
I am not sure what your last point means, the module does not work with fetching pages/content? Our A\B package hooks into the loaded content event in the CMS to determine if a user should be/are included in an A\B test, any loads of content under test via the API should trigger the A\B test logic as long as the load of said content triggers a LoadedContent CMS event. Could you clarify the behavior you are seeing?
Thank you so much for your quick response. Really appreciate it! Happy to have a call about this if you'd prefer? However, in the meantime I will come back to you on your points:
Page vs BlockSo ultimately we are testing at page level, but, to be able to test at page level we have to publish a new block on an unpublished (draft) page to begin the test.We cannot start an A/B test if we simply edit an existing block. Let's just say we wanted to change one word on the page. We have to create a new block paste all the content into it and change the specific word. It would be so much easier if we could just edit the existing blocks.
No approval on challenger versionThe challenger page can only be in draft and not published. Therefore we are showing a draft page to customers without having the relevant approval. From what i can understand of your comment this can be avoided - can you explain how?
A/B testing not working on Fetching Contentso the A/B package does 'hook' in to loaded content. But when we try and begin a A/B test on a page that fetches content it doesn't work.
Probably best to have a call about this if you get time?
The behavior you describe is correct for Pages vs Blocks since edits to blocks on a page is not considered an edit to the page itself. The page object in question has no updates to it's data since the block that has been edited had it's data changed, but the reference to the block that the page has is still the same. That is the same reason why editing a block on a page does not trigger a publish action on that page, only on the block. Is testing the block you edit not possible?
For content approval, the approval process just has to happen before starting an A\B test on the content. So the last step in the approval process will put the content in a Ready to Publish state, instead of actually publishing, just create and start an A\B test.
I think I understand the last part, you are programmatically fetching data for the page and loading it in? This is the same scenario as the block since we have to A\B test versions of a page object there would be no change in the page object if that code does something different our testing package has no way of determining what the version B should be. Our package can only test edits to the IVersionable content saved within the CMS.
I am not generally available for calls, but if you think the package is doing something in error feel free to open a support ticket. Our support engineers should be able to direct you to the correct resource to handle your issues.