How to have a conversation about something you can’t measure
“How do I know if my product team is actually doing well?”
The CEO had called me in to help figure out whether they had the right number of product managers. But within the first hour, the conversation shifted. The real question wasn’t headcount. It was whether the team was delivering. How could we tell? What should we be looking at?
I had to tell the CEO what I’ve told dozens of leaders before: there’s no clean answer. No dashboard that tells you “your product team is a B+.” No DORA metrics equivalent for product management.
My answer understandably received pushback. Engineering had velocity metrics. Sales had quota attainment. Why couldn’t product use something concrete?
Fast forward six months. By the end of our engagement, the product development team was operating noticeably better: shipping more consistently, communicating more clearly, making decisions with less thrash. But we never found the metric. We didn’t need to.
That’s the paradox of product team health. You can’t put a number on it. But you can talk about it.
Why measuring product teams is so hard
In product, we measure outcomes: conversion rates, retention curves, CSAT scores. We can build dashboards for everything our products do.
But measuring whether our product managers are performing well as a team? That’s a different problem entirely.
Sometimes an idea takes months to reach the market because the team iterated thoughtfully. Sometimes the best product work never ships at all. It dies in discovery because the team learned it wasn’t worth building. Both of those can be signs of a healthy product team, but neither shows up in a dashboard.
It’s not because we haven’t tried. It’s because measuring product team performance is hard. The signals that matter are cultural, contextual, and specific to each company.
So instead of a scorecard, I use a set of discussion prompts. The conversations usually happen with the people closest to the work (product managers, designers, engineers). But some of these questions also belong in a different room: with the CEO who wants to know how the team is doing, with finance, with operations. The people in the room change. The questions don’t.
Those closest to the work already have intuitions about what’s going well. They just haven’t had the space to articulate it.
The four lenses and their underlying questions
I organize these prompts around four lenses for product team health. They cover the full cycle of product work: how teams gather information (data and user understanding), how they collaborate internally to ship (ownership), and how they coordinate with the rest of the company (communication). Weakness in any one area tends to create problems that show up elsewhere.
- Using data: Healthy product teams make decisions with data in the room. Perfect analytics aren’t required. What matters is whether data is consistently part of the conversation, and whether everyone trusts it enough to use it.
- Understanding users: Strong product teams have built systems for staying close to users, not just occasional practices. Research flows somewhere useful, and findings connect to future work.
- Team ownership: Gone are the days where product decides what to build, design figures out what it looks like, and engineering makes it. All three roles need a say in the direction. High ownership shows up when everyone feels invested in the problem — when a QA engineer pushes back on a feature idea because it doesn’t actually solve the underlying issue, or when a PM doesn’t need to detail every last requirement because the team already has shared understanding.
- Cross-departmental communication: Strong communication allows stakeholders to feel invested in product success and reduces busywork for PMs. It comes from shared language, effective async communication, and processes that make alignment easier over time. More meetings won’t fix it — back-to-back syncs aren’t sustainable anyway.
Different product cultures will emphasize different lenses. A data-driven culture might invest heavily in the first. A customer-obsessed culture might prioritize the second. Start with whichever one feels most urgent for your team right now.
Each lens has two parts. The observable signals are examples of what “good” tends to look like in that area. They’re meant to spark your own thinking, not a checklist to follow religiously. Bring the questions into the conversation, whether that’s a 1:1, a team retro, or a meeting with your CEO.
Using data
Healthy product teams make decisions with data in the room. Not because data always has the answer (it often doesn’t), but because it’s always part of the conversation.
It comes down to whether the team reaches for data when making decisions, whether they trust what they find, and whether there’s enough shared understanding of metrics that people aren’t talking past each other. A perfect analytics stack isn’t required.
Observable signals to watch for:
- How often are decisions driven by data versus gut instinct, seniority, or internal politics?
- Is there shared language around metrics across product, engineering, and stakeholders, or does everyone define success differently?
- When a product decision is made, can the team articulate what they’d measure to know if it worked?
Discussion questions:
- Think of the last three significant product decisions your team made. What drove them?
- What would it look like at your company if data was consistently in the room, even when it doesn’t give a clear answer?
- Where does your data create more confusion than clarity? What’s one thing you could change about that?
Understanding users
Two things matter here: how often the team talks to users, and whether what they learn actually changes anything. For research to make a difference, it has to be easy to find, easy to apply, and shared widely enough that it doesn’t live in one person’s head. Plenty of teams do research. Fewer do it often enough, and fewer still let it shape what gets built.
Observable signals to watch for:
- Is there a clear path from a product idea → research → documentation of insights → connection to future planning?
- Does engineering participate in user interviews? How often?
- Can anyone on the team point to a decision that changed because of something users said?
Discussion questions:
- Walk me through what happens to a user insight after a research session. Where does it go, and where does it get lost?
- What would it look like if engineering felt as close to users as product does?
- What’s one thing your team has learned from users that surprised you, and how did you use it?
Team ownership
High ownership feels different from low ownership. It’s whether people feel like partners in what they’re building, or order-takers executing someone else’s vision.
You can usually sense ownership levels within minutes of joining a team meeting. People say “we’re building” or “they want us to build”. Engineers ask clarifying questions about the problem or just wait for specs. When a launch goes sideways, the room goes quiet, or everyone leans in.
Observable signals to watch for:
- Does the relationship between engineering and product feel like a partnership or a handoff?
- How much time does a PM spend detailing requirements versus co-creating direction with the team?
- When something goes wrong, does the team investigate together, or assign blame separately?
Discussion questions:
- Describe a recent project. Did it feel like a team built it, or like someone designed it and others executed?
- What does product-engineering partnership look like at its best here, specifically?
- Where is ownership clearest on your team, and where is it murkiest?
Cross-departmental communication
Product doesn’t build in a vacuum. Sales needs to know what’s coming so they can set expectations. Marketing needs lead time to craft messaging. Customer success needs to prepare for support tickets. When communication breaks down, launches feel chaotic, stakeholders feel blindsided, and PMs spend half their time in status update meetings instead of doing product work.
Strong cross-departmental communication shows up when other teams feel informed: when they understand what product is building and why. More meetings and longer emails don’t get you there. When it works, alignment gets easier over time.
Observable signals to watch for:
- Can sales, marketing, and customer success accurately describe what’s coming next, without asking a PM?
- When a feature launches, do other teams feel prepared, or blindsided?
- Does stakeholder feedback have a clear path back to product decisions, or does it disappear into inboxes?
Discussion questions:
- Think about your last major launch. Where did communication break down, and where did it work well?
- What do stakeholders misunderstand about how product decisions get made? How does that create friction?
- What would it look like if every team felt informed about the product roadmap, not just notified?
After the conversation
You won’t walk out of any of these conversations with a scorecard. That’s the point. What you’ll have is sharper language for what you’re noticing, and a few signals worth paying attention to.
Resist the urge to turn all of this into a project plan. The value is in the conversation itself: the places where you agreed, the places where you didn’t, and the things someone noticed that you hadn’t articulated before. You may have identified a place or two where you want to start measuring. Take it slow.
If the conversation got stuck on “we don’t have good answers to this” — that’s the answer. That’s where to focus next.
Don’t push for consensus. Surfacing disagreement is as useful as finding alignment. It gives you an opportunity to dig deeper into what the organization values from its product managers.
A few prompts for product leaders to reflect on during these conversations:
- What surprised you?
- Which lens felt most urgent for your team right now?
- What’s one signal you’ll watch for over the next 60 days?
Come back to these conversation prompts in six months. Or sooner, after a reorg, a leadership change, or a strategy pivot. The answers will be different.
What the CEO got instead
Remember the CEO who wanted to know if the product team was doing well? We never got the requested metric. But by the end of our engagement, we didn’t need it anymore. I could walk into a planning meeting and sense whether the team was aligned. The CEO could read a product brief and tell whether the PM understood the problem. We could feel the difference.
That’s what these prompts are for: not a score, but a sense.
You won’t get a number. You won’t get a letter grade. You’ll get shared language for what you’re trying to become.