metrics

Originally posted July,2012

Then in secret, Chezhin and Callahan happened. I’d say this particular shot would have maybe made Harry raise his eyebrow. Or better yet, he probably would have said; “nice going kid, but come up with something original and stop trying to emulate me” with a possible added, “now get away”. either way they continue to teach and educate far beyond to simply anyone who is willing to find them. it’s just so simple. Harry can come right off a text but he can speak to you even so, in the most profound way in his images. Yep, it’s all been done. everything. How often is he bastardized with unknown intents. but it’s only photography. it’s all been done, i suppose (correction).

Pentax K1000, Arista EDU 100 (B&W)

D76(Kodak), Fomacitro,Fomafix, Photo-Phlo

©2012auxiliofaux

Please join us this week for a webinar focused on learning more about Tumblr engagement. (Sorry, no snacks will be provided, so please provide your own as above.)

Kenyatta Cheese, Co-Founder of Everybody at Once, and Jenn Deering Davis, Co-Founder of Union Metrics, will sit down to talk about Tumblr, brands and how (and why) to foster engagement on the platform.

The webinar will be on Thursday, May 9th at 2:00 p.m. EDT.

Sign up here

Kenyatta is part of Everybody at Once, a company working on audience development and social strategy for media, entertainment, and sports. You may have seen his work on the very popular Doctor Who Tumblr for BBC America.

Jenn is co-founder and Chief Customer Officer of Union Metrics, the company that makes Tumblr’s preferred analytics application. Jenn holds a PhD in Organizational Communication & Technology from UT Austin.

During the webinar, Kenyatta and Jenn will talk about what goes into a successful Tumblr campaign, how to measure engagement, improve your content, and more. And we’ll share a coupon code for a month-long free trial of Union Metrics for Tumblr analytics at the end.

See you Thursday!

It’s dangerous to assume that numbers tell the whole story. It’s better to think of data not as a smoking gun, but as a trail of breadcrumbs. Metrics can point you toward problem areas or alert you to a potential issue that you might not have otherwise noticed.
—  We definitely agree with this statement in a pretty good article about metrics pitfalls: 5 Measurement Pitfalls to Avoid.
Tech dad

It is the day his son starts kindergarten. Tech dad shows the boy a graph of how the number of likes on Facebook photos of him waned, from hundreds at his birth to a piddling few of him playing in the park at age 5. “Here is the decline of your worth as a person in one chart” he says. “Now what are you going to do. What now. Have a good day at school. Make lots of friends.”

Metrics are a combination of art and science. The trick is to inspire action around the goals you set.

There’s an elephant in the room when it comes to “innovation.” And it’s an ironic elephant given that we’re all so hooked on data analytics, a/b testing, and getting metrics for anything and everything. Yet we all throw around terms like creativity, breakthroughs, and disruptive innovation. Companies eat up this stuff–they’re fully on board. Innovation is going to shape the future. Sure–if we track and shape it.

According to McKinsey, more than 70% of corporate leaders tout innovation as a top three business priority, but only 22% set innovation performance metrics. The gap is problematic. Why aren’t more companies measuring innovation? Because innovation is nebulous. Definitions differ. Expectations vary.

Read More>

Social Media is Bullshit

So there’s this, written by BJ Mendelson and published by St. Martins Press.

The premise of the book is that social media friends and followers “mean nothing to you and your business without old-fashioned, real-world connections.” Or, as the book site would have it:

We live in a world where it’s far easier to make money telling people how to get rich using the Internet than it is to actually get rich using it.

Which, I guess, would be fine if getting rich was the only metric one used for measuring their social media efforts.

Otherwise, I haven’t heard anyone ever say that real-world connections are no longer needed. Just the opposite, that our digital social lives help expand our ability to make real world connections. 

Anyway, I’ll be adding this to my reading list. I assume the argument is more sophisticated than little blurbs would have it. Besides, it’s always good to hear from a contrarian, even one who leverages his 770,000 Twitter followers to write a book and throw a snarky title on it. Hat tip to the marketing department on that one.

Meantime, here’s an interview with him and Andrew Keen, another Internet contrarian, via Techcrunch. And here’s a free chapter from the book. – Michael

Bonus: Trying to figure out your social media ROI? There’s a conference in New York for that.

KPIs for Social Media According to Executives Worldwide - eMarketer:

“Advanced” metrics had generally seen the most growth. For example, engagement—the top KPI—had jumped 32% in the past two years, while sentiment tracking showed year-over-year growth of 38%. …

Still, web traffic as well as followers, fans and group size—simple and relatively useless figures—ranked second and third, which… was “slightly disconcerting.”

The simplest and most important dashboard for early stage startups.

This blogpost is part of a small series of posts that cover the basics of startup metrics. My personal goal is to help early stage startups more effectively, but also avoid repeating myself too much in mentoring programs.

If you are working on a software product in early stage of its lifecycle this series might be useful for you.

Other posts in this series.

If there are special topics you’d like me to cover please ping me via @andreasklinger. Also if you consider them useful please be so kind and tweet this article.

Early Stage Startup Metrics

One of the most common pieces of advice is that in early stage you want to focus on retention. Because retention - in the end - is a function of customer happiness

retention = f(customer happiness)

It’s simple: If people are not happy with your product and/or gain no benefit by using it they will most not continue using it.

And automatically cool visualisations like retention cohort tables come to mind.

While those tables look nice and give you this nice fuzzy feeling of doing something useful with your time, they are unfortunately usually a bit useless until you get to the point of optimising on-boarding conversions of new users.

Please don’t get fancy too early.

This kind of visualisations are great when you have too many users and need to get a bird eye view. When you need to an abstraction. B2C mobile apps often have this problem already in early stage. But most SaaS companies (or companies still in beta) i worked with don’t have this problem until they reach product market fit.

20% changes with 20 customers won’t tell you much apart of the basics of standard deviation. A personal relationship to this customer will tell you his story. Keep them close.

If you treat your paying customers like percentages and meaningless numbers they will do the same with you.

The one “dashboard” i recommend almost all early stage startups is one they already have. Let’s start using it.

As long as your customer list fits on one or two pages, you should have a list of all those customers. (And i am pretty sure you already do have this in your backend)

Add activity information to this table. Core KPIs that tell their activity level (or at least their last login date). Color highlight it based on activity. And put someone in the team in charge of making sure everyone stays on green.

dummy mockup is dummy

Seriously? That has nothing to do with metrics, or?

“That’s the kind of advice gives someone who charges money for consulting? Make him draw complex graphs or bring the pitchforks.”

This dashboard so embarrassingly simple to implement, i was even scared to publish this post. But it is - and keeps being - my most common recommendation for a custom dashboard.

I don’t know about you. But I don’t necessarily need numbers. I need information i can act on. I want product insights - simple ways to use my database information to create actionable insights.

Most abstractions like graphs, retention cohorts, aarrr funnels and so on are great to aggregate information as soon as you have too much of it. Before that they just do one thing, create an abstraction layer.

For myself i found out this is even useful in a bit later stage: If you have only 0 - 100 new signups don’t hide them neither. It’s the same game on a higher level. Their first month is the one month that decides if you will keep them or not. Keep someone in charge of making them stay green.

Put one person in charge of keeping all customers on green. Most likely that person is you.

Your job as a product manager/founder is it to keep in contact with all those customers. And as soon as someone moves to orange contact them via email or skype.

Find out what’s going on. Now they are still in their decision process. Now you still can convince them to stay. Now you might even get useful information for product changes.

Churn is not happening when a user unsubscribes from your service or stops paying you. This is just when you notice it. Churn happens when a user stops using your product. You want to jump in when a customer is at risk, when he might stop using your product.

As long as you have a few dozen paying customers you can contact each one personally. There is no excuse. Don’t treat them as numbers, if you don’t want to be treated the same way.

Customer Happiness Index

If last activity of your customer doesnot mean much for your product (eg login might not mean that he found/did something) you might want to focus on deeper core activities.

Usually people create a Customer Happiness Index. A single number that combines all done activities to a sum, while giving each of those activities a weight.

By doing that you can eg. say that logins are less important to you than purchases. You can even aggregate those numbers in groups and through that see problems in certain segments of your user-base.

But to be honest, as fancy this is, i hardly ever need this.

Not until i need to aggregate those numbers. Eg show the customer happiness of a certain customer segment.

But until then - in my personal experience - i usually only see 1-3 core activities in a product. (eg in a project tool - number of projects/total, todos-closed/week, active-team-members/week) And normally i tend to simply recommend adding those numbers to the table and be done with it.

Commercial break.

One product i want to highlight at this point is Intercom. They basically offer this as a service, but add manual and automatic messaging. Additionally they write one of the best startup product blogs out there

But as long as you need/want to save a bit of money. The table mentioned above is only a few lines of code away.

until then, get your product moving,

Andreas

Daily UX Crash Course: 2 of 31

When you start a new UX project — before you design anything — you need to understand your goals. Two of them, to be specific. Everything you make is based on these two goals and nothing is more important to your success as a UX designer:

User goals and business goals.


(If you missed Lesson #1, you might want to read that one first!)

User Goals:

Users always want something, because they are people, and people always want something. Whether they are trying to get laid on a dating site, looking for sneezing pandas on YouTube, or stalking old boyfriends on Facebook, they want something. They might also want to do something productive (or so I am told).

(We will look at methods of user research in upcoming lessons. For now, just assume we know stuff.)


Business Goals:

Every organization has a reason for creating a site or app in the first place. Typically it’s money, but it might be brand awareness, or getting new members for a community, etc.

The specific type of business goal is important. If you want to show more ads your UX strategy will be a lot different than if you want to sell products or promote via social media.

These things are often called “metrics” or “KPI’s” by the business-y folks.


Align the goals:

The real test of a UX designer is how well you can align those goals so the business benefits when the user reaches their goal. (Not the other way around!)

YouTube makes money via ads, and users want to find good videos. Therefore, putting ads in the videos (or on the same page) makes sense. But more than that, making it easy to search for videos and find similar videos will get users to watch more, which makes YouTube more money.

If the goals are not aligned then either users can get what they want without helping the business (lots of users, no success) or the users don’t get what they want (no users, no success). If YouTube made you watch 60 seconds of ads for every 30 seconds of videos, they would die a quick, painful death. Ain’t nobody got time fo’ dat. But a few seconds of ads is a small price to pay for those sweet, sweet sneezing pandas…

****

Tomorrow we will look at the 5 major aspects of turning good goals into a good UX strategy...

A Primer on Startup Metrics – Which Analytics Tool to pick.

As a analytics consultant I am working with a lot of startups and help them in the topics of metrics, analytics, retention and growth. Most of the questions i get are based on a few misunderstandings.

Eg. What kind of analytics tools you should use depends what you are trying to do with those tools. They have (more or less) specific purposes.

Disclaimer: This post focuses on web products - but you could apply the same concepts for mobile.

The Segmentation of web analytics

So let’s get into this…

First: Please lower your expectations

If you are hoping to add one tool and be ready to go, i need to stop you right here. Please readjust your expectations.

There are a few hard truth facts we need to accept:

All tools are good for a few things, but tend to suck for most of the other - the trick is to know what you are looking for.

Also: Any tool you will integrate will most likely be: not optimal for all of your usecases, not specific enough, most likely be broken or incorrect a good amount of your time and contradicting every other tool you have.

But that’s ok. If you know how and what to use them for.

Let’s roll…

I personally split usage of analytics into two axis:

  1. exploration / accounting.
  2. external / internal

Exploration vs Accounting

Exploration are those ad-hoc questions.

Exploration are all those quick questions people come up with and are very often related to some intuition we have.

  • eg “What feature is most used?”
  • or “Are users who use feature X more likely to upgrade to plan Y?”

Conversion Rates, Landing Page Optimizations, Funnel Optimizations, Viral Loops, Quick questions and insights are they typical examples of usecases of exploration.

Accounting are the metrics we track over time.

FYI the term accounting was coined by Eric Ries as innovation accounting covering more metrics than the typical product metrics, but the term kept sticky in normal web analytics as well.

If you have ever kept record of KPIs in your excel sheets (eg for investors) that’s a typical report we want for accounting.

But it also includes reports here we tend to use several times a week.

Cohort Retention Tables, Customer Happiness Reports, Traffic Referral Reports or Revenue Reports are typical examples for accounting reports.

About Data integrity

In case of exploration it usually doesn’t matter too much if we have only little historic data, or that we need to jump through a few hoops (aggregate from several tools, export to excel/R, etc). We need as much data as we need to invalidate or validate our hunch/gut-feeling or current objective. Very often this insights can and will be challenged by the team and need further verifications.

These insights usually lead to experiments (eg a/b tests).

In the case of accounting it is very important to have historic data and data integrity is very important - we can’t really tolerate too much noise in the data we are using for accounting. We can’t really tolerate broken reports nor can we have our team not trusting those reports.

This insights usually lead to strategic decisions. (eg switch sales focus)

While it is very useful to distinct between these two areas we need to keep in mind that in reality the lines are a bit blurry.

The second axis i tend to look at is external and internal analytics.

External vs Internal Analytics

  • External Analytics covers all public facing pages and all passive actions by users (eg views).
  • Internal are all actions done by (persistently hopefully) identifiable users, that potentially reflect if your product provides value to them.

When people speak of “product insights” they tend to mean that internal view of your product.

Many product have a very clear distinction between external and internal - eg. SaaS tools provide a value that i need to register for.

Sometimes it’s a bit blurry - Some products provide usecases to non-identified users as well - eg. content pages like pinterest or 9gag where a huge amount of the product is public and most of the visitors value is provided passive.

In a nutshell

Which Tools

There is roughly a million of analytics tools out there. I will focus in this article on the most common ones for web products: Google Analytics, KISSMetrics and Mixpanel.

Google Analytics:

GA is great for everything external. Especially traffic analysis and referral optimization work perfect in GA.

I am a huge Google Analytics geek and to cover it properly would be at least 3 own blog posts.

If you are new into it here are three blogposts i would recommend you to read:

The most common usecase for GA is referral optimization. It enables you to understand what sources bring you the most valuable traffic and where you should double down (as said above learn how to use weighted sorting).

If you use GA for all external stuff you will be very happy. For product internal insights GA tends to be useless.

The main reason is that GA doesn’t think in users and events. It thinks in visits and sessions.

You can force GA and focus more on visitors, you can segment down to the pages that are product internal, people that are registered and you could look mainly at events. But even if you do all of that then you will have a hard time using that data.

GA wasn’t made for product insights in first place.

GA is awesome for product-external exploration and very good for a lot of product-external reporting/accounting (eg weekly referral/traffic/seo reports).

But it stops being spot-on useful where most products actually start. (activation/engagement)

Ad Mobile: I am speaking obviously mainly about GA for web here. The GA team currently pushes their mobile analytics tool. Everyone who has used will notice this weird feeling that the tool wasn’t meant for this originally in first place. But given their current development speed I wonder if i will need to come back here in half a year to update this part.

KISSMetrics and Mixpanel

Both - KMS and MXP - come from the same main concept. They think in “people” and “events”.

Both tools are great and to cover them well would need several blogposts by themselves so please excuse my brevity.

At first they look very similar and i am pretty sure you could substitute every feature of the one in the other. But to me they both tend to have a bit different focus. They have been build by teams with a bit different values.

Mixpanel is awesome for product-internal exploration and good for product-internal reporting. It’s the tool i would pick if i focus more on product improvements (eg engagement). Personally I also have better experience with its stability.

KISSMetrics on the other hand feels like it has one foot in the product-external world.

It’s revenue tab makes it very easy to see what sources brought customers of what kind of life time value. You can easily see what your customers where doing and so on.

Of course MXP is catching up and offers a revenue tab as well. But nether the less KISSMetrics would be the tool i’d pick if I would focus a lot on revenue improvements (eg through content marketing) while trying to improve the product itself (especially as a SaaS startup)

Actually KISSMetrics fits so well as a hybrid between product-internal and external usecases - especially for content marketing - i wouldn’t be amazed if they come up with a specialised content marketing tool or somehow else integrate with Google Analytics to have a easier angle to the market.

Specialised tools

There are a lot of really good specialised tools and again not possible to cover all of them here. My personal four favorites are

And of course there are many more.

The advantage of specialised tools is that they understand (or expect) enough of your product internals that they can provide value insights right out of the box.

The big problem with all tools

Most tools are integrated through push events. Basically your site will send an event through javascript or your backend and let their service know.

There is a big problem with this approach.

Data Integrity

Sooner or later your integration will break for a day or two. Because someone on their or (more likely) your site made a mistake.

Sooner or later you will change how you track certain events and your comparison with your historic data will no longer work.

Did you ever see different numbers in different tools? Eg different amount of registrations between MXP and your database? or Revenue in KMS and in your stripe dashboard? Well that’s what i am speaking about.

In case of exploration this usually tends not to be a big problem. Anyway the data i work with tends not to be optimal (sample noise, dataschmutz, skewed, variance etc) and I am just looking for anchor points for future experiments.

In case of accounting this is a big problem. We cannot really have our revenue reports incorrect. This is unfortunately where the fun ends.

And there is a other problem…

Focus on product internal aspects

Very often product-internal reports tend to be quite specifically tied to your product.

  • If you run a market place you want to report improvements on both sides separated.
  • If you sell not to people but to companies you want to aggregate your users into those company buckets.
  • If you have different levels of users you might want to treat them different (eg drill down your best paying or engaged segment)

To figure the right metrics is a blogpost by itself but to in a nutshell you want to tie your product assumptions to quantified results.

All of the mentioned segmentations above are doable in some tools mentioned, but it will require you to jump through several loops.

And given the fact that very often data integrity tends to be a problem the logical conclusion is to sooner or (if possible) later build those highly specific reports by yourself.

Work with your database for accounting

When it comes to accounting i would highly recommend to work with your database.

As a rule of thumb: The more you need to zoom in - the better you work based on your own database.

Depending on the level of complexity you need you can:

  • Export to excel
  • Run a script that sends the most important numbers
  • Build a simple dashboard

In most cases i would recommend to start with the first one and do it for the first months. And then start step by step automating the most time consuming report parts to scripts and from scripts you will move to a simple dashboard view over time. Usually i would recommend stopping here.

If the dashboard itself turns out to be too time consuming (eg most likely because you put too many expectations into it) try using a tool like chart.io

If you pre-run into performance problems move the analytics to a own server and run the scripts only once a day.

Doing that you can introduce reports like Customer Happiness Index and Health Dashboards (red/green lights (or kpis) per customer or segment)

Disclaimer I am not advising you to build your own analytics tool. I not recommending to measure non-logged-in data. I advice you to segment your existing database data for reporting. There is a big (costly) difference.

I won’t go into dashboard design right now - but if you are interested in this topic please let me know via twitter happy to write a blogpost about it.

So to answer the question – which tool would i recommend

These are the swiss army knifes. Depending on stages i would recommend other tools additionally or instead.

If i have to decide between MXP and KISSMetrics i would - usually - go with Mixpanel (but as explained above it depends a bit on the specific product).

Hope this blogposts helps you a bit to get clarity about metrics even though i couldn’t cover all aspects nor all tools.

If you are still unsure which tools to pick i would recommend using Segment.io as a wrapper for your API calls. That way you can simply replace your tools whenever needed.

If i can help you anyhow with your analytics setup - let me know - i am doing weekly open office skype sessions - feel free to hit me up on twitter

Until then - measure/build/rock

Andreas