tinker on

dreaming up a web that works.

Tag: metrics

Correlation or Causation?

by Santosh

Came across two apt examples of the use of data to demonstrate correlation and causation.

On Orbitz, Mac Users Steered to Pricier Hotels“, WSJ.com

Orbitz has found that people who use Apple Mac computers spend as much as 30% more a night on hotels, so the online travel agency is starting to show them different, and sometimes costlier, travel options than Windows visitors see.

Orbitz executives confirmed that the company is experimenting with showing different hotel offers to Mac and PC visitors, but said the company isn’t showing the same room to different users at different prices. They also pointed out that users can opt to rank results by price.

Orbitz found Mac users on average spend $20 to $30 more a night on hotels than their PC counterparts, a significant margin given the site’s average nightly hotel booking is around $100, chief scientist Wai Gen Yee said. Mac users are 40% more likely to book a four- or five-star hotel than PC users, Mr. Yee said, and when Mac and PC users book the same hotel, Mac users tend to stay in more expensive rooms.

Private Schooling Myth Debunked“, The Age

Children who attend private primary schools don’t perform any better in NAPLAN tests than their peers at public schools, new research shows. It was the children of a healthy birth weight, who grew up in higher socio-economic circumstances in homes filled with books and had mothers who didn’t work long hours who performed best at NAPLAN.

Children who weighed less than 2.5 kilograms at birth, achieved ”significantly lower” test scores, especially in grammar and numeracy, with the researchers suggesting low birthweight correlated with longer-term developmental delays.

Children whose parents had completed year 12 had higher test scores across all subjects. Students whose mothers worked long hours did worse in all tests except numeracy, yet the working hours of fathers had no impact on test results.

”One explanation for this may be that children of young ages typically spend more time with mothers than fathers,” the authors said.

Like the ideal product practitioner – we’re primarily interested in discovering causation over correlation. You’ll often find instances where two variables show correlation but aren’t linked causally. The latter article wants to say that low birthweight is not the reason behind lower scores on standardized tests. On the other hand, the kind of hours that parents spend with their children do have an impact on how children perform.

The first article on Mac users booking pricier hotels makes the distinction blurry. Is owning a Mac simply correlated to my taste in hotels? Is Orbitz right in assuming that Mac users are less conscious about price to value? Or is there more to it than meets the eye? How you translate this information into your product tells your users a lot about your brand and the difference you see between correlation and causation.

How Sticky Growth Models Work

by Santosh

What really matters is not the raw numbers or vanity metrics but the direction and degree of progress. – Eric Ries, ‘The Lean Startup’.

Most online services can be cast as having a sticky engine of growth. At it’s heart is the question – “Is the experience rewarding enough for new users to return?”. A model built around this question can help you determine the direction and degree of progress for eCommerce services, creative communities, Saa’Services and many more business models.

Three measures come together in this model – Retention Rate, New Customer Rate and Growth Rate. Each measure has it’s own story to tell. If you’ve read ‘The Lean Startup’, you’ll also learn that the model can reveal if your venture has figured out how to leap forward consistently.

With this honest a metric at hand, you won’t lose your way. I’ve found the model handy in most of the projects that I’ve worked on. Below is an interpretation of the model and how the three measures are derived.

Sticky Growth Model (Image)

With the model above,

1. Retention rate: “Customers Retained in Current Period” by “Total Customers”.

2. New Customer Rate: “New Customers” by “Total Customers”.

3. Churn Rate: “Customers you failed to retain” by “Total Customers”.

4. Growth: is the difference “New Customer Rate” – “Churn Rate”.

5. Total Customers: coming into a period are the Customers Retained in the previous period and New Customers you will engage in the current period.

Key scenarios shown in the test sheet (Google Docs Worksheet) that will help you grasp how the model works,

* When Retention Rate is 100%, Churn Rate is zero.

* When New Customers Added = Customers Retained in the same period, Growth is 100%.

* When Retention Rate is zero, Growth is negative for that period.

* When New Customers Added is zero, Growth is less than or equal to zero depending on the Churn Rate.

Thanks to Mitesh Bohra, CEO at Savetime.com for lending his time to whet iterations of the model. Do leave your feedback in the comments, especially if you have a different interpretation to share.