Part 4 – Introduction

Over the last few posts we have been looking at how to set up and fund business technology teams – teams that are directly linked to business functions and provide the technology component of the business as an integral part of the business rather than something provided by a separate IT group. These teams are funded by the business on an ongoing basis in order to deliver business outcomes.

This is a very different model from the traditional one where teams are formed to deliver projects. Project teams typically are on the hook for specific project deliverables. Long-lived business technology teams are on the hook for actual business outcomes. So the way you measure those teams is also very different.

A new way means a new way of measuring

Traditional measures of project success, if applied to teams delivering business outcomes, will give a very poor view of the team’s actual performance. As an example, in project-based organisations, it is very common to track change requests, the number of changes from the originally agreed scope with few changes being seen as desirable.

In an outcomes-based system we want the teams to innovate and change direction to achieve the best possible outcome rather than stay fixed to the original plan. So these teams typically have very high levels of change requests and look very bad on that measure even if their actual outcomes are far better than project teams who stick to the plan regardless of the outcome they deliver. Outcome-based teams need to measure actual outcomes – actual value delivered.

The Three Types of Value

Before we can measure the value a team delivers, we need some idea of what value of each of the pieces of work they do should be, so that when they are delivered we can test to see if that value was realised. This is non-trivial as value means very different things for different teams, and even for different pieces of work within a team. One piece of work might be a customer-facing feature that drives revenue through new subscriptions, another might be a back end upgrade that saves time for the team itself and maybe unlocks some new features. There is no one size fits all measure of value.

Generally, value comes in 3 basic types –

• Direct customer or business value – a new feature that directly adds value to the customer experience or directly impacts the business through efficiency, cost-saving, etc.
• Enablement – something that adds no value directly but unlocks future value. Often system upgrades have significant enablement value attached.
• Learning – Something that adds no value directly but gives us valuable information about the best way to proceed in future. Prototypes often have a lot of learning value attached.

Of course, it’s not quite that simple, things generally aren’t one type of value or the other, they have a bit of all three in them. A database upgrade may enable a lot of future value through new features but it may simplify maintenance (business value) and improve performance (customer value) plus give us valuable insights into the future direction of the tech stack (learning).

The Hypothesis – We expect…

Regardless of the type of value, we identify it through the Hypothesis. A hypothesis is a fancy way of saying guess, or assumption. Our hypothesis is that when we do this particular piece of work, we will get a particular outcome or outcomes. If we upgrade the database we will save 20% on maintenance costs over the next 6 months. If we release this feature we will get 10% better customer conversion. And so on.

The hypothesis sets out the value you expect to get from that piece of work. If you can’t come up with a good hypothesis for something it may be an indication that there isn’t any value associated with it. Or that the piece of work is poorly articulated and re-thinking it might make the value clearer.

We will measure this by

Once we have a hypothesis that sets out the value we expect, the next thing we need to do is work out how we will measure that value. This may not be easy. You may not be able to measure the value directly but have to rely on indirect or proxy measures.

There are some things to watch out for when measuring value.  The first and most important is to beware of single metrics. Measuring one thing almost never gives an accurate measurement of the real value of something. You need a suite of measures to tell the whole story. If you measured someone on just typing speed you would encourage fast but very inaccurate typing (much like mine). You need to measure both speed and accuracy to get a true picture. Likewise just measuring new customer acquisition isn’t a full picture – are you attracting them by giving away a lot of introductory offers – what’s your cost of acquisition? Are they profitable customers or are they low value and high maintenance?

Are your measures leading or lagging? Do they give you an early warning that something will happen or can you only tell what happened after the event? Ideally, you want leading indicators as these act as steering signals. They will tell you whether you are on the right path.

Effectiveness

Now that you know what the value is and how to measure it, you are in a position to start measuring the team delivering that value. Essentially the question you are trying to answer is whether the team is effective. Note that I didn’t say “efficient”. I can be very efficient at delivering things that don’t actually add any real value. The more efficient I get, the faster I throw money out the window by building things that don’t work.

Traditional measures like cycle time give you a view of efficiency, not effectiveness. 

We typically measure effectiveness through the flow of value rather than the flow of work – how much value is the team delivering per unit time? Is the money we are spending having the desired impact?

If it isn’t, is it because the team is not building the right things (effectiveness problem) or that they are building the right things but too slowly or too expensively (efficiency problem)?

You want to build the team’s effectiveness first. Make sure the team is working on the right things. That they understand and measure the value of what they do. Once the team is effective, we can start building efficiency. Working on streamlining processes and tools to improve the flow of work. 

A team that is both effective and efficient is an amazing asset.

Read Part One: Is it time to reimagine your IT department?