AI OnAI Off
A/B test on a block leads to unexplainable results
Do you use output caching on the site? It could be that just the page is cached and it shows the same version that first got cached for all users.
Jan 09, 2018 17:09
Maybe a stupid question but is the participation %'ge set to 100% of the participants?
Jan 09, 2018 22:20
Thanks so far !
I checked with our back-ender and we do not use output caching. Also, I made sure to set the participation to 100% in order to get the 50/50 split.
Edited,
Jan 10, 2018 10:23
Hi Diederick I think you might have found a bug. It seems A/B test does't take visitor groups into account.
I have just added a block in a A/B test to a page with a visitor group that will never match. However impressions are still registered for the block.
Please can you raise a bug using the raise a bug option here: https://world.episerver.com/support
Workaround
In the meantime you can A/B test the page the block sits on with two versions of the block as you described both personalised using the original visitor group. This should give you the 50/50 split plus only show the block at certain times of the day.
Jan 12, 2018 17:52
Hi David, that would explain a lot ;) I'll look into the workaround you described.
I would gladly raise a bug however i don't see that specific option on the support page. When i want to Register an incident it tells me to log in even though i am already logged in.
Jan 15, 2018 10:28
Hi Deiderick
I have reported this as a bug and will get back to you when its publically visible.
David
Jan 16, 2018 17:25
Hey,
I'm currently running an A/B test on a specific block on our 'Contact Us' page. This block contains a link (the only link) to our contact form. I'm not using the full functionality of Epi A/B testing; I'm only using it to make sure 50% of our visitors sees variant A of that block and 50% sees variant B. Both variants of the block look identical to the user, but link to two separate pages. Variant A of the block links to Contact Form A. Variant B of the block links to Contact Form B. According to the Epi statistics for the running A/B test, the two variants of the blocks are being shown 50/50. So far so good.
Now comes the thing that puzzles me. Since this block is the only place we link to our contact form, and since the two variants of the block are being shown 50/50, both variants of the contact form should have the same number of views (provided our users navigate there ONLY through the block on the Contact Us page). However.. Google Analytics disagrees. According to GA, during the time the A/B test has been running (since the 13th of december), Contact Form A has had roughtly 5 times more views. GA also tells me that nearly 100% of these views are from the Contact Us page, so it's not like people are navigating to Contact Form A directly. All traffic comes through the block that should in half of the cases link to Contact Form B...
Any ideas what could be going on here? I'm starting to lean towards the explanation that Epi is not actually showing the two block variants 50/50 at all. It may also be related to the fact that the block that is being A/B tested is shown conditionally - only to visitors that view the Contact Us site during evening hours and during weekends. Could it be that the visitor group conditionality is interfering with the A/B test? Another weird thing is that the Views counter in Epi keeps increasing, even during the time of day that this block should not be shown to visitors.
Thanks in advance for any insights!