tinker on

dreaming up a web that works.

Tag: Analytics

Correlation or Causation?

by Santosh

Came across two apt examples of the use of data to demonstrate correlation and causation.

On Orbitz, Mac Users Steered to Pricier Hotels“, WSJ.com

Orbitz has found that people who use Apple Mac computers spend as much as 30% more a night on hotels, so the online travel agency is starting to show them different, and sometimes costlier, travel options than Windows visitors see.

Orbitz executives confirmed that the company is experimenting with showing different hotel offers to Mac and PC visitors, but said the company isn’t showing the same room to different users at different prices. They also pointed out that users can opt to rank results by price.

Orbitz found Mac users on average spend $20 to $30 more a night on hotels than their PC counterparts, a significant margin given the site’s average nightly hotel booking is around $100, chief scientist Wai Gen Yee said. Mac users are 40% more likely to book a four- or five-star hotel than PC users, Mr. Yee said, and when Mac and PC users book the same hotel, Mac users tend to stay in more expensive rooms.

Private Schooling Myth Debunked“, The Age

Children who attend private primary schools don’t perform any better in NAPLAN tests than their peers at public schools, new research shows. It was the children of a healthy birth weight, who grew up in higher socio-economic circumstances in homes filled with books and had mothers who didn’t work long hours who performed best at NAPLAN.

Children who weighed less than 2.5 kilograms at birth, achieved ”significantly lower” test scores, especially in grammar and numeracy, with the researchers suggesting low birthweight correlated with longer-term developmental delays.

Children whose parents had completed year 12 had higher test scores across all subjects. Students whose mothers worked long hours did worse in all tests except numeracy, yet the working hours of fathers had no impact on test results.

”One explanation for this may be that children of young ages typically spend more time with mothers than fathers,” the authors said.

Like the ideal product practitioner – we’re primarily interested in discovering causation over correlation. You’ll often find instances where two variables show correlation but aren’t linked causally. The latter article wants to say that low birthweight is not the reason behind lower scores on standardized tests. On the other hand, the kind of hours that parents spend with their children do have an impact on how children perform.

The first article on Mac users booking pricier hotels makes the distinction blurry. Is owning a Mac simply correlated to my taste in hotels? Is Orbitz right in assuming that Mac users are less conscious about price to value? Or is there more to it than meets the eye? How you translate this information into your product tells your users a lot about your brand and the difference you see between correlation and causation.

Audience and Product Experimentation

by Santosh

A startup is essentially a combination of market, product and team bound together by vision. With these conditions – knowing why your product is working and what the bottlenecks are is key to making inroads into a market. A great habit is to constantly test for the details – did the recent features additions do what they were intended to do, or did they take you in the wrong direction? Is your product on track to realizing the projections I’ve made?

If you are new to product experimentation – here’s a past anecdote to help give you a clearer idea of how this worked. 2006 was the year where India was waking up to the idea of bringing the box office online. Even though user’s did not know what to expect when booking a ticket online, BookEazy’s engineering team extended themselves to testing each and every feature they’d built and released against our market assumptions.

The kind of questions that were asked included tradeoffs between searching for a movie ticket by movie, show time, or by theatre. Which ones did user’s prefer? We successfully validated that a) user’s preferred a combination of searching by movie and by theatre on the desktop, and b) by show time when coming in via the mobile including a host of other questions like these. The learnings were reflected in how we designed our home page and ticket booking workflow.

Quickly and definitively learning facts about your audience, monitoring how these facts change over time will get you to a stage where you can experience growing user delight daily and differentiate your product from your competition.

The tradeoff in studying cause and effect in user behavior is the effort taken to complete the experiment. If the experiment is too hard to setup, you probably won’t execute it. That’s a missed opportunity to learn about your audience. For instance, the ability to A/B test is key to validating or invalidating assumptions we make about the ideal behavior expected from user’s for every product feature. And yet, I don’t know of any open web frameworks or template engines that allow you to simply drop-in a plugin or module to instantly implement A/B tests without the pain of wrapping individual links. Since you will need to engineer your own A/B test layer, or go with an external tool, you might choose to not implement split tests in favor of early time to market.

If you’d still prefer to apply some degree of experimentation, consider starting with Cohort Analysis. The key idea is to organize your user’s into clear groups and then compare their ideal behavioral measures over time. For instance, you could divide your new user’s by their sign up month  – as Fred Wilson has shown here, or existing user’s by release date to validate that your release has taken you in the right direction. The key difference between analyzing in cohorts versus a simpler analysis tool is that cohort will tell you how a specific population has done over time. If you’ve added an onboarding wizard recently to your product, your cohort analysis should tell you how the more educated population has done when compared to previous populations. They should ideally yield a far greater lifetime customer value over time when compared to other populations. You can visualize a simple report on Cohort Analysis 101.

If cohort analysis isn’t immediately possible, enlist 3 to 5 key measures that you think are best representative of your business value and segment your audience. To be very concise, two measures of tell a significant story – How many users use your product actively? How many users return to your product? Start there and work with a web analytics tool such as Google Analytics or Mixpanel tied in to your product and the necessary discipline to regularly track the impact of releases.

Week on Week comparison of Avg. Visit Duration - Google Analytics.

Week on Week comparison of Avg. Visit Duration – Google Analytics.

Other posts on experimentation,
Product Development and Experiments, Mixpanel blog.
How We Reduced Our Cancellation by 87.5%.
User Activity Streams and Cohort Metrics.

Lipikaar gets popular across languages and applications

by Anjali Gupta

The recent data from Lipikaar shows that we have gathered users across the spectrum.

Key highlights:

No one language accounts for more than 20% of users. A year ago we had Hindi and Punjabi dominating our charts.

The top 10 languages used by Lipikaar users are – Hindi (19%), Arabic (17%), Punjabi (13%), Marathi (10%), Gujarati (8%), Telugu, Malayalam, Bengali, and Tamil.  Urdu and Kannada are tied at the 10th spot.

On the Applications front, we have users across 300 Unique Software Applications!  Users have typed in the above Indian languages on 300 different Windows Applications. The most popular one Microsoft Word accounts for only 3%!

The top applications – Microsoft Word, Excel, Access, Internet Explorer, Acrobat, Firefox, Chrome, Outlook, Notepad, PowerPoint, GoogleTalk, Yahoo Messenger, PhotoShop, and so on.

Some of the new entrants that are being actively used with Lipikaar are Google Earth and iTunes.

After powering the PC and websites, we’re gearing up to power the mobile phone with Indian languages.  Do send us your ideas. Write to me if you would like to include Lipikaar with your software or mobile application.