Why you should layer in solutions beyond tools
I had just finished reading a great book about A/B testing products and had all sorts of ideas about how we could improve our feature testing practices at SpotHero. I realized we weren’t always reaching statistical significance on our tests. Furthermore, I saw the opportunity to begin using A/B testing to better articulate the value of what the product team was contributing to the business.
The challenge was that I wasn’t running the majority of the tests – the product managers and designers responsible for the feature changes were generally leading the charge. So I had to educate them on what the problem was, how to perform tests properly, and why it mattered. And then I implemented the infrastructure to support the change.
I didn’t rely on training alone to get the team there. I explained the situation and my concerns and conducted a little training on what good tests look like. I built out a slide template for us to fill out with every experiment to track our hypothesis. We did a call with our A/B testing tool vendor to get additional guidance. We started meeting weekly to review test performance, discuss hypotheses, and coach each other on whether we were still following best practices. We dug into our tooling that we had in place and started using it more rigorously. And finally, we documented what our expectations were for using the tool.
It sounds like a lot. But for the changes that matter, more is more.
The rule of (at least) threes
I implemented a lot of small pieces of infrastructure to make this change happen.
When working with clients today, I coach them on the rule of (at least) threes. Implement (at least) three different types of infrastructure to encourage the behavior change to stick. Keep each individual piece of infrastructure as simple as possible.
I see infrastructure falling into eight major categories:
- Tools. The software a company uses in order to do product management.
- Templates. Resources that the product team uses to jump-start their work.
- Documentation. Explainers to set expectations around how product work is done.
- Training. Investing in the team to enable them to do product work better.
- People. Making sure the right people are in place and have the time to support product work.
- Automation. The connective tissue between pieces of infrastructure to reduce manual labor.
- Workflows. Clear steps to follow for making product work happen.
- Meetings. Moments in time for people to get together with clear purpose and steps.

As you can see, infrastructure goes far beyond tools. Infrastructure is all the things we can do to make people change the way they act in a company. When it’s important for a change to work, it helps to layer on more types of infrastructure.
More infrastructure is (almost always) better
Product operations implements the right infrastructure to create the behavior change we want. That behavior change creates our desired product culture.
The rule of (at least) threes works because each separate piece of infrastructure builds on the others. It makes it more likely that the rollout plan can touch every element of a change management model, making it easier for everyone to make the shift to the new behavior.
The more radical or important the behavior change, the more layers of reinforcing infrastructure might be needed.
I’ve rarely seen a team implement too much infrastructure to make a change happen. The mistakes I’ve seen more frequently have been not enough infrastructure or overly complicated infrastructure (that’s an article for another day).
Imagine some alternative scenarios from SpotHero with less infrastructure:
- Had we learned what good looks like, but weren’t communicating well with each other, we would have launched tests simultaneously that interfered with each other.
- Had we started tracking our hypotheses well but continued using the tool incorrectly, our data would have been garbage.
- Had we learned how to use the tool better but not documented it, we would have struggled to onboard new team members correctly, or would have forgotten the right thing to do ourselves.
- Etc., etc.
Our vision for the product team included a data-driven culture, which requires good testing practices. I wanted to make sure the changes stuck and that the whole team could be effective over the long term. A/B testing needed to become core to our culture.
For people to permanently shift their behavior, we need to put multiple layers of infrastructure in place. Note how many layers and types of infrastructure I used:
Layer | Type |
Taught A/B testing best practices | Training |
Slide template | Template |
Call with our vendor | Training |
Meeting to review test performance | Meeting, training |
Improved tool use | Automation, tooling |
Writing up tool usage expectations | Documentation, workflow |
Not all of this was permanent – as we got better at running tests, the meeting frequency dropped. We updated the documentation and template as we figured out what did and didn’t work. We iterated on our infrastructure to make sure it was continually working for us. (No, we did not a/b test any of this. Our sample size would have been too small!)
Each layer of infrastructure reinforced the other layers. It helped turn the behavior into a habit.
Product infrastructure is only one part of one step in the stack
Implementing infrastructure is only one step within the broader product operations strategy stack. Before choosing and implementing the infrastructure, establish a vision, roadmap, and strategy for the product organization. Once the infrastructure is in place, measure what has been implemented to make sure it’s helping achieve the desired outcomes.

Similarly, when implementing the infrastructure itself, use that same product mindset. Conduct user research to evaluate the situation, come up with multiple ways to solve the problem at hand, and continuously iterate on the solution as the organizational needs change.
With the introduction of the experiment-driven infrastructure at SpotHero, we saw an increase in experiment velocity. More importantly, teams started talking a lot more about the impact of feature changes. It meant that we were able to track our progress towards our OKRs more easily and share those wins with the team. Engineers appreciated the visibility into their work and the whole team morale got a boost.
Culture is hard to change. The best way to do it is to make sure you’re reinforcing the desired behavior in multiple ways. By using multiple layers of complimentary infrastructure, I was able to make a shift in our product culture. Next time you’re trying to make a change that lasts, check your list – did you follow the infrastructure rule of (at least) threes?
Thank you to the awesome people who gave this one a pre-read: Matt Willman, Julian Cheah (#OpenToWork), and Brian Weber (#OpenToWork). Your feedback helped me make sure I wasn’t burying the lede.