In his book, The Lean Startup, Eric Ries writes about the processes we used at IMVU to get Minimal Viable Products out there as experiments to observe the customer adoption as the prototypical Lean Startup.

I stepped in after Eric went off to evangelize Lean Startup principles to the industry. I remember an experiment we did on helping our “whale” customers that were gifting a lot of products to their friends and followers, but the gifting was a cumbersome, one at a time process that we thought we would make better by allowing them to give to groups. So, we did an experiment to make it easy to give to multiple people and collected some data. Much to our surprise there was no noticeable improvement in terms of gifting. So, to follow our methodology, we were ready to scrap the experiment and revert back to the old code (no use in adding functionality that doesn’t bring value)…

However, it didn’t sit right with me as logically speaking, it really should’ve made a difference if I understood our customers right (which I thought I did having spent a lot of time chatting with them).  I decided we should split the data out and differentiate between customer type to see the impact of the change on behavior of whales, non-whales, etc.

Subgroup A1 - up ~48% over two weeks prior
Subgroup A2 - down ~68% over two weeks prior

Low and behold, gifting was way up for customers that were whales but way down for the typical non-whales and in aggregate, the net change was not noticeable. It turns out by making the experience to gift to many easier, we had made it a bit harder for those wanting to gift to just one. Instead of abandoning the experiment, we then allowed for both through the easiest path for each. When customers gift, they spend more credits and increase engagement – both valuable to the business. Had we trusted the first view of the data and had allowed our decisions to be purely data driven, we would have completely missed this opportunity.

My take-away is that in being data-driven, don’t turn off your brains or ignore your gut instincts about the customer and the business. When it comes to data, as with many things – trust but verify.