Throwback – Standing Up For the Unmeasurable in a Data-Driven Design Process by Tobias Ahlin

A talk condensed into vital points

Tobias Ahlin is the Principle Design Engineer at GitHub, and former Experience Design Director for Minecraft and a previous Product Designer for Spotify. With a haul of knowledge of the ever-evolving digital industry, Tobias Ahlin’s teachings have direct and applicable truths that are provoking and aspirational.

Design has become survival of the measurable, instead of survival of the fittest. Hitting all the targets, but completely misses the point.

The tension between the love of being informed, by data and the negative aspects of being driven by data -is the core of Tobias Ahlin’s talk.

The point of data is to help us to see more clearly –this is not always the case. Metaphorically, data can be described as depicting one square inch of a large painting. Stating the colors, textures, and cracks of the little examined piece. Though these facts represent interesting details, the whole point and feel of the painting can be lost in the analysis. Simply put, ‘precise’ data does not mean ‘accurate representation’.

‘Portrait of Adele Bloch-Bauer I’ painting by Gustav Klimt.

Our current culture highly believes in performance management systems. Such as S.M.A.R.T (Specific, Measurable, Achievable, Relevant, Time-related) or O.K.Rs (Objectives and Key Results) –Giving us a perverse incentive. The act of measuring can cause unintended consequences. We forget the non-measurable aspects of the design process.

SMART goals can easily overtake core unmeasurable values. Image of framework from BiteSize Learning.

Unmeasurable Aspects:

-Is it delightful?

-Is it inclusive?

-Is it beautiful?

-Is it simple enough?

How Do We Keep Our Values in the Design Process?

We can change our measuring sticks and include more outcome options in our hypotheses. This will allow for more than one conclusion, and prepare an openness for different and more accurate results. For clarity, Tobias Ahlin mentions a hypothetic example…

‘By featuring our offer at the top of our app, our sales and revenue will increase.’

The above hypothesis needs revision, as it has a single short-term conclusion in mind.

‘By featuring our offer at the top of our app, our sales and revenue will increase, unless retention decreases.’

-If the retention decreases, revenue will decrease in the long term. The revision makes the statement an actual hypothesis because the possible outcome can vary. More variables in a hypothesis will bring more true data. If a hypothesis can only be proven correct, it is not a hypothesis -but a trap.

Perverse Intentions

The problem with any data performance metric is; that it will eventually be either gambled, manipulated, or diluted of its original value- in congruence with goal setting. It is the inevitability of the ‘Rat Effect’.*

*The Rat Effect refers to the outcome of the Great Hanoi Rat Massacre of 1902. A solution to a rat infestation and bubonic plague threat, that did not go as planned. The Colonial Government offered any enterprising civilian one cent per dead rat. To minimize the overflow of rat corpses, a rat-tail would represent the dead rodent. Shockingly enough the infestation increased, and tailless rats were often sited. The entrepreneurial spirit spread as far as the countryside, where multiple rat farms were established.

Retention is Goal Gold

Retention is a long-term metric and is generally seen as the ‘golden one’ as it is the proxy for quality. It is, however, hard to detect meaningful changes in retention. One measuring device can be to remove a single feature in your product or service and test the elasticity of the feature. Retention is usually not elastic.  

Sticking to core values can be profitable. Because often, they target long-term metrics. So refrain from selling your soul for quick profits. Instead, be smart and moral -the good guys, apparently do, have the last laugh.


To keep both the measurable and unmeasurable variables viable in the design process, try to measure differently. Use a mixed measurement method. Drive your process with qualitative data, the immeasurables. And to ensure that you don’t regress, keep yourself informed by the quantitative data as you launch or roll out. Measure less, measure thoughtfully, measure better. Removing metrics will make the work culture healthier and overall better.

“Don’t forget what brought you into design.” –Tobias Ahlin

Watch Tobias Ahlin’s Talk below. To view more talks, subscribe to

For more updates and inspiration from Tobias Ahline, follow him on Twitter @toviasahlin or view his website where he shares his designs, tinkering, teachings, and other projects.

Photo of Tobias Ahlin
Discover all the design events and conferences by Design Matters! Seize the opportunity to learn from an amazing cast of international designers in a series of events - in June in Tokyo & Online, and in October in Copenhagen & Online.

Did you enjoy the article? Share it on