Significance of Statistically Significant Results in A/B Testing

You are running an A/B test (or multivariate test) but are in a hurry to make a decision to pick the winning page. Should you pick a winner based on initial few days of the data before your tool has actually declared a clear winner? This question comes up quite often during the conversations with the clients.

I always warn against such an approach and advice to be patient and get statistically significant results before pulling a plug on an underperforming variation o declaring a winner. I wanted to share few graphs with you to illustrate my point.

The following example is from a test that I am currently running.
First Two Weeks: If you make a decision without completely getting through the test, you will pick “Yellow” as the winner while declaring “Blue” a loser.


Month Later: There does not seem to be a clear winner.


Few More Days Later: Seems like “Blue” is trending higher. Should you pick “Blue” now?


Finally: We don’t have a clear winner yet.


As you can see, picking a winner or dropping a loser in the early stages of test, without having a statistical significant result, would have been a wrong decision.

Comments? Questions?


Looking to fill your Web Analytics or Online Marketing position?

Post your open jobs on Web Analytics Job Board
--------------------------------------------------------------------------

Komentar

Postingan populer dari blog ini

Bikin Hilang Fokus !! Berikut Potret Natural Mikha Tambayong Saat Kenakan Baju Tidur

Cara Berpakain dan Duduk Cupi Cupita Bersama Hotman Paris Dikritik Netizen .. Kenapa Ya ?

Walaupun Terpaut Usia , Berikut 7 Potret Keharmonisan Bunga Zainal Bersama Suami