Anybody tried using cohort.js? Or something else?
Interesting, I certainly check it out. What is your experience with it?
Right now we distribute new users evenly into 8 layered testing groups on sign up, run tests with own template logic and collection, track with google analytics and pull the data back into mongodb for analysis via the aggregation framework to than create actionable information.
We’re using Optimizely, without issue.
I have not used anything yet. And there is only one package on atmosphere, so I am trying to get community opinion what we can do.
Sounds really complicated. For now, I just want to start something simple. Button A round, Button B square, what user would like.
It is surprising there is practically nothing on atmosphere for that
Check out Optimizely, it’s dead simple. You can select an element like a button, apply a different style to it for variation #2 or #3. The only challenge is figuring out where to put the Optimizely JS calls, but that’s not that difficult.
Doing A/B testing is nothing easy. Not the programming part, but the statistic part. That’s why there’s nothing on atmosphere I guess.
I am really missing something I guess.
10 users came, 5 were shown round btn, 5 square.
3 out of group A clicked on More, 2 out of B clicked on More. Save it in collection, do median and mean, add something like region, if you have anything like age and gender too.
That’s practically it. Why stats would be complicated in here? Meteor is soooo suitable for that task with dynamically changing templates and elements.
Or even do not do anything inside an app, just send it to Google analytic. Google will sort 'em
Maybe something more elaborated would be not that easy, however this is what I would need for the start. I guess it is me only.
Well, you need more than 10 users to know if round “is better” (whatever is better means to you). For example, Optimizely offers a Sample Size Calculator that take statistical significance inside the game.
Big players that are focused on these kind of testing have so much better precision and techniques that you should go with them, since you have a already solid product in hands. If your product is not solid, probably you are wasting money/time doing A/B testing. You can take a look what I’m talking about here: http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf
and here: https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
First, when I am talking about 10 users, I am not talking about 10 users in my population or even set. It is an example.
Second, sometimes it is good enough to accept not just alpha 0.05, but 0.1 or even 0.2. Sometimes even 0.5 can you give a very good idea, if you know what I am talking about
It all depends on what I want to get.
We are using Raisemetrics, it’s really simple. Previously we have also used Optimizely and Unbounce with great success.