I spoke with various Developer Experience product managers about how they measure success–what KPIs they use and why. It became apparent that the responsibilities of each DX team shaped the KPIs they use. There are four potential domains that developer experience teams are responsible for, with different KPIs for each domain.
|The Four Domains|
|Get people to the developer portal||Convert people on the developer portal to customers||Retain developers||Operate the API gateway or program|
|Primary KPIs||Increasing quality people landing on the dev portal||Conversion rate||Retention rate||Uptime and latency|
Of the four areas, everyone I spoke with was responsible for the middle two sections. Depending on the company, the team also had one of the responsibilities on the ends, but no team I spoke with was responsible for all four domains. While not everyone tracked the same KPIs, I will highlight the ones that I found to be the best fit. When discussing the KPIs below, I define them as primary KPIs and leading indicators. I do this because each domain has one KPI that is the ultimate measure of success. Because the KPI can be hard to measure or slow to gather, many teams use leading indicators as secondary metrics to support their work.
Get people to the developer portal
Not every DX team is responsible for marketing to developers, but plenty of them have responsibility for developer evangelism. As Cristiano Betta explained to me, if you’re responsible for traffic to your website, then you’re really responsible for getting potential customers to the point of sale. Given that, your KPI should be focused on increasing the number of quality visitors that arrive on your site. This requires defining your target audience, figuring out how to target them, and serving the right content at the right time. A good analytics program is critical at this point.
Convert people on the developer portal to customers
Once you have people on your site, it’s time to convert them and, just as in ecommerce, your main KPI becomes your conversion rate. The KPI is easy to understand but hard to break down, and so leading indicators are especially helpful here. Bandwidth uses usability testing, time to first hello world, and site analytics together as leading indicators. The combination of qualitative and quantitative measures here seems especially helpful.
Once you’ve got your customers, it’s time to retain them. As with the conversion piece, all the product managers I spoke with talked about this domain in one way or another. Some phrased it as “supporting the developer community” or “working with customers”, but it comes back to retention. The main retention rate is straightforward enough, but leading indicators are less concrete. Possible metrics include adoption and use of tools, an increase in volume of API calls, and pull requests on open source projects. Returning to the ecommerce analogy, it’s equivalent to size of shopping cart.
Operate the API gateway or program
The least common responsibility of a DX team is actually managing the operations of their developer program. In one case the product manager explicitly told me that this is a DevOps role, and it was not the primary focus of any DX team I spoke to. However, for those who need to operate the program as a part of their responsibilities, standard DevOps metrics apply: uptime, latency, etc. The general consensus was that the more mature the company, the less likely this role would be part of the DX responsibilities.
While each specific area has corresponding KPIs to match, the one piece that touches the whole spectrum is developer satisfaction. Given that it’s an emotional goal, it’s very hard to measure with anything concrete. Some use net promoter score, but that gets fuzzy—it’s hard to separate out satisfaction with your developer experience program versus the company overall. Slack creates a few leading indicators for developer satisfaction by keeping an eye on documentation quality and adoption of tools. They’re also great at the most important part of measuring satisfaction– listening and talking with their developer community, whether through online tools or events, which allows them to really capture the qualitative side of this metric. Others mentioned deflected questions—questions that get answered through a knowledge base or support flow without requiring a person to intervene to help the developer find resolution.
Choosing your KPIs
Defining your KPIs is an opportunity to define the scope of your team’s responsibilities. Take the time to connect across your organization to figure out who is taking care of your customers and at what point; once you know your responsibilities, the KPIs should follow.
This post originally appeared on Pronovix on August 24, 2017.