Why creating metrics is so hard: the human and technical challenges

In the modern landscape of business, business metrics are no longer optional for success. The combination of increased competition and a rapidly changing world make measurements critical for preventing missteps and driving growth. In the first part of this series, we discussed the ways and areas good metrics can help steer a company, as well as the gaps in communication that make metrics such a challenge to implement. In this second part, we want to provide an overview of the technical and human factors that complicate the adoption and use of business metrics.

Technical Challenges

1. Data Silos

Business structures and technology have become frustratingly complex. Even small businesses will lean on a multitude of third party applications, sites, and services for their everyday operations. Each of these entities generates its own data and has processes for making data available. The frequent occurrence of this excess of data sources and applications has led to its own class of products, tools specifically for combining data or connecting these services together. In this complex framework, data is siloed by default. Useful metrics that require data from disparate sources are nearly impossible to attain without doing a tremendous amount of work connecting those data silos, or combining data from very different sources in a robust and reproducible way. Combining data across sources presents all sorts of challenges and potential misalignments, for example:

• data type mismatches
• missing data in either system
• broken connecting data
• system configuration differences (timezone, numeric accuracy, encoding)
• duplicated or overlapping data
• different levels of granularity
• syncing schedules

2. Data Freshness

The timing and freshness of data is crucial for making the right decision at the right time. Making business decisions based on stale data can cause missteps. That means that our systems need to not only communicate with each another, or at least put all their data in one place, but that all of this has to happen quickly, and often. Depending on how you’re moving or transforming your data, you may need to carefully time different actions based on the syncing schedule of the data source.

3. Scalability

Once you tackle the issue of connecting a dozen disparate sources of data in a meaningful and timely way, continued operation and success bring new data sources and new metrics. Sooner than every company may hope for, issues of scale and performance show themselves and start to hinder a business’s trajectory. The systems designed to connect, consume, clean, and transform the data will start to break or fail as the amount of data and the complexity of your metrics grows. The solutions to this are varied, but usually involve the significant investment of resources for storage, processing, and maintenance. Careful thought and planning must be given to the infrastructure supporting your metrics, and having a poorly designed or underpowered system can prevent the calculation and delivery of metrics. In the long term, a poorly designed and supported system may yield incorrect or misleading metrics that cause lost revenue and costly mistakes.

4. Maintenance & Delivery

So now that we’ve combined our data, it’s fresh enough to keep us informed, and we’ve managed to create a system that can grow as our data does. Now we have the final step of storing, maintaining, and delivering it. Metrics are useless if no one ever sees them, but exposing and disseminating metrics is a challenge all on its own. Each company has different levels of technical skills, or wants different metrics, or likes to consume data in different ways. Setting up reporting that is accessible enough for use by business stakeholders and multiple departments, while allowing it to still be maintained by the data team or other engineers, is difficult and requires a lot of consideration. Some of the biggest challenges that come with this area are less technical, and more about the people we’re designing and building these systems for and with.

Human Factors

As if the technical hurdles for creating reliable metrics weren’t tall enough, companies will often find themselves in a position where even if the technical aspects of their systems are performing, they meet human resistance for a variety of reasons. This organic resistance can come from a lot of different angles, but two of the biggest are cultural resistance and interdepartmental friction.

1. Cultural Resistance

Change is a constant, but acceptance of change is not. Introducing new systems or metrics can be met with resistance from employees, especially if they feel it may disrupt their current way of working. People can often be creatures of habit and so some systems, even if they work imperfectly, are difficult to shift away from.

Less technical users may be overwhelmed, confused, or discouraged by programs or systems that are presented as too “tech” or code forward, while technical data people might resist tools that appeal to business teams but don’t allow the data team to govern metrics effectively.

2. Interdepartmental Friction

Even in small companies, there are competing goals and priorities from different areas of the business. The communication between departments and groups within an organization makes or breaks a company. We know we need metrics for every arm of the business, but the languages each department speaks can be wildly different. A “customer” to the marketing team may be a completely different definition than for the data team, the customer service team, the revenue team, etc. Maybe the marketing team thinks of a “customer” as anyone whose email you have. Maybe to the revenue team it’s anyone who paid you more than a dollar. These different definitions and ambiguities can cause friction between departments, especially if definitions and communication are not clear and agreed upon. If you’re presenting a metric that involves the number of customers and the CFO is expecting a “customer” to be someone who purchased something, but you’re presenting with the marketing number of anyone who has signed up, your numbers will be very different and convey very different things. The issue is compounded if the CFO doesn’t realize the nuance, or is assuming a different set of criteria, and making decisions from the wrong place.

—

The root challenges of creating effective metrics span from very technical to very human. To drive any effective change in a business, a metric must be well-designed in the abstract, technically well-implemented, and culturally well-deployed and received.

We built Preql because we’ve encountered all these obstacles firsthand, and from every angle. We do the hard, technical work of connecting disparate data sources in a scalable, maintainable way, but we’ve also built a platform to make managing metrics definitions and reporting delightful for both business stakeholders and data teams. Get in touch to see how we can help carry some of the load for you.