The hardest part of any initiative is almost never the software itself. Usually the real complexity lives in everything around it: the structure, the logic, and the processes you have to build so the software actually earns its keep.
Picture yourself and your fellow decision-makers in a room, needing to make a data-driven call. Which department to double down on, where to pull the budget from. Right now, could you actually pull that off?
Why bother thinking about data collection?
If you run a technology team, you've almost certainly asked yourself at least one of these questions:
- How many features am I shipping per sprint?
- What's my team's average development time per issue?
- How many bugs get opened per feature we build?
- What's the rework rate across my department?
If you were able to answer all of those, congratulations. You already have a solid foundation for making decisions.
And if you couldn't, don't panic. As I write this series, I can't answer them either, yet my team keeps shipping.
So if the team is already delivering, why pile on another layer of complexity in the pipeline?
Here's the thing: in software, the small stuff compounds. Tiny details can quietly snowball into major consequences in the final product and over the long run.
If I don't know the average time per task, how do I know how many features I'll deliver to my client over the next two weeks? If I don't know the bug rate per feature, how do I know the quality of what we ship is where it should be?
Sure, these are questions you can try to answer by gut, right? The day-to-day feel of the team. But more often than not, our gut isn't fully in sync with reality. Lean on it too hard and you'll overload the team with too much work, underuse it with too little, ship something shaky thanks to a creeping bug rate, and so on.
And that's exactly where data-driven, intelligent decision-making earns its place.
What data should you collect?
To figure this out, I went down this path: what do I want to answer today that I currently can't?
Being honest with myself, the questions I listed above already cover a good chunk of what I can't answer right now.
And even if you don't have the questions to answer yet, agile methodologies come loaded with common-sense metrics that can point you in the right direction (they pointed me, too). In this piece, I'm sharing a bit of my own thought process so you can draw inspiration from it and build your own. A few of them:
- Lead Time: the average time your team takes to deliver a request, from the moment the issue is created to the moment it ships to production. It's the full wait a customer sits through before getting the feature.
- Cycle Time: the average time your team takes to deliver a request, but measured from the start of active development to production. Unlike Lead Time, here we're timing one specific stage (design, development, and so on).
- Throughput: how many requests the team can deliver within a given window of time.
- WIP (Work in Progress): how many requests are in flight at any given moment.
- Bug Rate: the rate of bugs found in production per delivered request.
- Rework Rate: the share of rework needed to fix bugs found in production.
- Defect Density: the density of defects found in production per delivered request.
- Customer Satisfaction: how satisfied customers are with what you've delivered.
- Team Satisfaction: how the team feels about deliveries across functions (knowing how the dev team feels about QA and design, for example).
Notice how many of these tie back to the questions I raised earlier. These metrics really are generalist, they're baseline assumptions of software development.
How do you collect it?
This is the question I'm still answering and building out, because the process isn't fully baked yet. But to give you a head start: these metrics can be gathered in all sorts of ways, such as:
- Project management tools (Linear, Jira, Trello, Asana, ClickUp, and others)
- Bug tracking tools (Sentry, Bugsnag, and the like)
- Performance monitoring tools (New Relic, Datadog, and so on)
- Satisfaction surveys with customers and the team
What matters is being able to collect the data consistently and reliably, so you can make decisions grounded in evidence rather than gut feel.
Wrapping up
At this point, the metrics I've settled on tracking are locked in, and the collection process is still being built.
In the next posts, I'll walk through how that process comes together, along with the early results at UEEK.
If you've made it this far, I hope this got you thinking about how to collect data across your own engineering pipeline, and how it can move the needle for your team and your business.