November Happy Hour will be moved to Thursday December 5th.

AI OnAI Off

A/B Testing Add In - production ready?

Steve Esson
Steve Esson
Vote:
 

Hi,

I have installed and tested the EPiServer.Marketing.Testing Add in:

EPiServer version 10.10.4

EPiServer.Marketing.Testing version 2.4.4

My experience is that this is far from production ready, is this the gerneral opinion here? Poor documentation and very buggy.

  1. Installing in an instance that uses WebAPI breaks the site - https://world.episerver.com/forum/developer-forum/Addons/Thread-Container/2016/11/ab-testing-installation/
  2. Kills performance, renders edit UI unusable. This was supposed to be fixed in 2.4.0 but clearly not the case. 
  3. This is a doozy - viewing another lanaguage variant of a page under test produces a 404! Another critical bug that was supposed to be fixed in 2.4.0, demonstrably not the case.
  4. Rendering thumbnail previews of Control and Challenger pages takes ~10 minutes!
  5. For the test I made i chose Landing Page as a goal, I have yet to trigger a single conversion in my testing. Simply doesn't work.

As you can see, this module is so broken it cannot possibly be ready for a production environment - it does not work. Perhaps the latest version + EPiServer 11 improves things. I think the 2.4.* QA team should hang their head in shame.

Questions re latest version + EPiServer 11

  1. Does this module require session state to be enabled?
  2. Is this module supported in a load balanced Azure App Service?
  3. I can't find the job that clears the generated preview thumbnails on a schedule (a png file is written to [root]\modules\_protected\EPiServer.Marketing.Testing\abcapture EVERY time the test is edited or viewed) - does such a job exists? If no, is there any sensible logic in the moduel to clear these image files?

Thanks,

Steve

#193866
Jun 06, 2018 16:08
Jason Masterson
Jason Masterson
Vote:
 

Sorry to hear you are having such trouble with the package. I would suggest creating a support case for the issues you are running into so they can be reproduced and worked on by the dev team. Obviously, if the problems you are running into had presented itself during the QA process the package would have been held up and not released so finding what is hindering the funtionality for you is essential to fixing it.

As for your questions:

  1. The session state is used for the thumbnail generation to mimic the user to see the draft version of the content, it is not used in the triggering of conversions or views.
  2. The module should work in a load balanced App Service environment. Tracking of the user through an AB test is done via a browser cookie so when a user views or converts the cookie is then updated so subsequent requests can take the appropriate action no matter what machine processes the request.
  3. In the latest version of the package thumbnails are no longer being saved to temp files, they are just generated on demand and shown in the UI. The previous versions that generated temp files should have removed the temp images upon closing of the test screens, but it seems like some of the issues you are running into may be preventing that normal behavior.
#193919
Jun 07, 2018 15:48
Evgeniy Muzyka
Evgeniy Muzyka
Vote:
 

Hi

I agree regarding point with thumbnails - load takes a huge amount of time, it's useless for blocks and doesn't render CSS styles well (phantomjs limitations)

Don't you have an option to disable it?

Regards

Evgeniy

#194898
Jul 05, 2018 17:22
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.