Audience and Product Experimentation

by Santosh

A startup is essentially a combination of market, product and team bound together by vision. With these conditions – knowing why your product is working and what the bottlenecks are is key to making inroads into a market. A great habit is to constantly test for the details – did the recent features additions do what they were intended to do, or did they take you in the wrong direction? Is your product on track to realizing the projections I’ve made?

If you are new to product experimentation – here’s a past anecdote to help give you a clearer idea of how this worked. 2006 was the year where India was waking up to the idea of bringing the box office online. Even though user’s did not know what to expect when booking a ticket online, BookEazy’s engineering team extended themselves to testing each and every feature they’d built and released against our market assumptions.

The kind of questions that were asked included tradeoffs between searching for a movie ticket by movie, show time, or by theatre. Which ones did user’s prefer? We successfully validated that a) user’s preferred a combination of searching by movie and by theatre on the desktop, and b) by show time when coming in via the mobile including a host of other questions like these. The learnings were reflected in how we designed our home page and ticket booking workflow.

Quickly and definitively learning facts about your audience, monitoring how these facts change over time will get you to a stage where you can experience growing user delight daily and differentiate your product from your competition.

The tradeoff in studying cause and effect in user behavior is the effort taken to complete the experiment. If the experiment is too hard to setup, you probably won’t execute it. That’s a missed opportunity to learn about your audience. For instance, the ability to A/B test is key to validating or invalidating assumptions we make about the ideal behavior expected from user’s for every product feature. And yet, I don’t know of any open web frameworks or template engines that allow you to simply drop-in a plugin or module to instantly implement A/B tests without the pain of wrapping individual links. Since you will need to engineer your own A/B test layer, or go with an external tool, you might choose to not implement split tests in favor of early time to market.

If you’d still prefer to apply some degree of experimentation, consider starting with Cohort Analysis. The key idea is to organize your user’s into clear groups and then compare their ideal behavioral measures over time. For instance, you could divide your new user’s by their sign up month  – as Fred Wilson has shown here, or existing user’s by release date to validate that your release has taken you in the right direction. The key difference between analyzing in cohorts versus a simpler analysis tool is that cohort will tell you how a specific population has done over time. If you’ve added an onboarding wizard recently to your product, your cohort analysis should tell you how the more educated population has done when compared to previous populations. They should ideally yield a far greater lifetime customer value over time when compared to other populations. You can visualize a simple report on Cohort Analysis 101.

If cohort analysis isn’t immediately possible, enlist 3 to 5 key measures that you think are best representative of your business value and segment your audience. To be very concise, two measures of tell a significant story – How many users use your product actively? How many users return to your product? Start there and work with a web analytics tool such as Google Analytics or Mixpanel tied in to your product and the necessary discipline to regularly track the impact of releases.

Week on Week comparison of Avg. Visit Duration - Google Analytics.

Week on Week comparison of Avg. Visit Duration – Google Analytics.

Other posts on experimentation,
Product Development and Experiments, Mixpanel blog.
How We Reduced Our Cancellation by 87.5%.
User Activity Streams and Cohort Metrics.