And all the tips and tricks I learned along the way
For the past 6 months, I’ve been developing a workshop to help teams create a product culture blueprint.
At the end of the workshop, I’ve finished by saying “you now have all the materials you need to write up a blueprint yourself!”
The reviews of the workshop have been positive, with comments like “I wish we had done this ~2 years ago as it can help create foundational alignment on ‘what is product management?’.”
I conduct a follow-up a few weeks later and have found that (almost) nobody has had the time to write the blueprint up themselves. The work we’ve done together goes undocumented and unused.
Having a well-articulated product culture is foundational for product operations. It creates alignment, helps with recruitment, and serves as the vision that product ops builds towards. Some companies with great blueprints include Netflix, Airbnb, Hubspot, and Duolingo.
I’ve been doing these workshops for free, so couldn’t justify writing up the blueprint for them after the workshop.
That makes blueprint writing a perfect use case for AI.
So I built the Product Culture Blueprint Drafter. It’s an AI tool to help take the conversations from these workshops and turn them into a usable working document for people to edit.
Here I break down more about why I chose this particular use-case, how I built the tool, and a whole bunch of lessons learned.
Automation is no longer optional
Claire Vo stood up at the Lenny and Friends summit this fall and announced that product management is dead. Ignoring the sensationalist title, she argued that AI means we can no longer do product like we used to.
If that’s the case, it’s up to us in product ops to lead the way. This bot is one part of my journey.
The goal isn’t to eliminate work but to get it out of the way. AI can draft documents, generate agendas, write updates, summarize meetings, create slides, and even explain product features. The trick is knowing what to automate. If it can be done faster and with 75% accuracy by AI, there’s no reason you should still be doing it yourself. Automation is no longer a strategy; it’s a baseline requirement.
I’ve been asking every product ops person I talk to about what AI use cases they’re enabling. The majority of folks are providing access to a chat-based LLM like ChatGPT or Gemini, sometimes with additional training.
Many teams are finding AI-enabled tools. These vendors help with customer insight analysis, writing release notes, and other familiar use cases.
Many of the strongest product teams I’m talking to are in the middle of building or have already deployed custom-configured AI tools. They are focusing on workflows that are unique to them and taking advantage of the data within their walls.
It’s time for me to level up my own AI skills. I called up John Haggerty, instructor of Become an AI Product Manager to ask for a little coaching (use code JENNY15 for 15% off the course*).
How I built it
The whole process took me about 4 half-days of work to get up and running. The most challenging parts were selecting the right focus for the tool and fine-tuning the model to get the output to be more than adequate.
Select the one job to be done
I almost made a gigantic mistake. My first idea was to build a product ops bot – upload my entire newsletter archive into the tool and let anyone ask anything product ops related.
John explained that AI isn’t a great multi-tasker, and you should pick one narrow use case for your bot. He gave a few other pieces of advice for how to choose the purpose:
- A repetitive task
- Something you do the same-ish way every time you do it
- Low risk – you don’t want a major crisis if it’s wrong
In essence, something that takes up a fair amount of time and would be valuable if automated away. If I were building the business case for it, I would be translating the value into people-hours and from there into a dollar amount.
The product blueprint bot fit this well because it has one purpose – to produce this particular type of document. The document has a well-structured format and so the output can be easily programmed. It’s a narrow use case, but necessary.
Most importantly, it’s valuable. I wasn’t able to write this document for my workshop participants before and had only written them for paying clients. Now, I can make sure that my workshop participants get the final product from our time together. I’m enabling something that I wasn’t able to do before.
Find your source material and analyze it
Once I had my use case, I needed source material. Luckily, I had a blog post, course materials, and examples available. And this is important – if you don’t have a clear point-of-view on what a great output looks like for your tool, you’re going to get a generic output.
I used AI to break down the supporting content into clear summaries and instruction sets. That became the underlying content for the prompt.
At this point I also had to think about what content I or my users would have available to upload into the system to generate the blueprint.
The transcripts from the workshops were gold mines. I could combine those transcripts with the outputs from the whiteboard. This would provide enough content for the tool to work its magic.
Build the bot
I chose to work with a custom GPT from OpenAI because those are the easiest to make publicly accessible; you might have a different use case and so pick a different company.
Just because the bot is focused on one task doesn’t mean it can only have one type of interaction. I mapped out the different steps that a user should go through when interacting with the bot.
I made it simple: someone would run a workshop outside of the tool and upload a transcript and supporting documents. The bot would then ask clarifying questions and produce a culture blueprint. (Perhaps one day I’ll build a bot to run the workshop as well)
I plugged in the descriptions of my process and formats that I had already created with the help of AI. I added additional prompt instructions to the configuration that I thought would be useful context.
Once I had something drafted, I used Prompt Engineer to clean up my sections and make everything cleaner. It added significantly more structure and enhanced the details of the instructions.
OpenAI makes it easy to add custom instructions to your bot. At this point I dropped the prompt in and uploaded some sample material to the other side that allows me to test the tool.
The initial output was not that detailed or interesting. I refined the prompt and tried again.
Eventually, my prompt hit a point where I was comfortable with it (for now) and I was ready to start alpha testing.
For alpha testing, I started running the prompt with new transcripts that it hadn’t seen before. I made sure the output with new material matched my expectations. From there, I’ve been running soft launches and getting feedback on the output from the companies whose workshops I ran in the fall.
My biggest challenges: depth and hallucinations
Building an AI tool like the Blueprint Drafter wasn’t without its hurdles. While I encountered many small challenges, there were two that stood out above the rest: getting the bot to be substantive in its responses and avoiding hallucinations.
At the beginning, the output of the tool was shallow. Responses were short and generic. There were a few things that helped fix this:
- Uploading sample material: My first pass I just used the prompt to describe what I wanted instead of uploading examples. I was afraid that the tool would try to copy the example instead of relying on the user-provided material. Once I uploaded an example the response depth improved and it didn’t copy the example.
- Demanding a certain amount of output: I added the following to the prompt to make my requirements clear: VERY IMPORTANT! Each set of descriptive paragraphs should be at least 150 words long.
- Emphasizing that outputs should be unique: I mentioned several times in the prompt that the output should feel unique to the company that is being discussed and shouldn’t have platitudes.
With these additions I started getting outputs that were sufficiently detailed and dense. This was a double-edged sword; the more I emphasized uniqueness, the wilder the outputs.
I started to be given hallucinations – content that wasn’t in any of the uploaded materials. The titles for each section should be extracted from a table I uploaded, but instead the AI made up its own.
As I iterated, I realized that there was another tool I needed – a tracking tool to make sure I could know what changes I had made to prompts and the issues with each version.
I built out a small table to track what the prompt was, what the output looked like with the sample material, and what issues I needed to address.
This allowed me to have some form of version control as I continually edited the prompt.
I was able to go back a version to diagnose what was causing the hallucinations and revert that one section to make the bot more reliable.
I also learned the hard way to write the prompt in my tracker and then copy it over. Occasional errors on OpenAI’s side caused page reloads and I lost my work on more than one occasion.
Build your own bot
As you take a journey through building your own AI bot, remember my key learnings:
- Make it good at one thing and one thing only
- Develop your own expertise and point-of-view before building
- Use a prompt engineer to improve your instructions
- Track your prompts outside of the AI platform
Focusing on these four learnings will help you avoid building a tool that outputs the generic AI content we’re seeing everywhere. It will encourage you to focus on creating something that helps you work faster.
Identify your own use cases and give this a go – the experience of building out the bot has helped me better understand what AI can and cannot do well.
If you’re interested in using the product culture blueprint drafter, you can get access here. If you’d like me to run a workshop for you to get a workable transcript, please reach out. I reserve a few slots each month to run these workshop with companies around the world.
Many thanks to John Haggerty for the coaching! If you’re looking for help with any stage of AI implementation, he’s got a wealth of knowledge that can help you integrate AI technologies into your products. Take a look at his course, Become an AI Product Manager, and use JENNY15 for 15% off.*
* This is an affiliate link.
Reading:
- Why is there so much hype around agentic AI by Curtis Michelson. A good primer on what makes an agent different from a chatbot or other AI-powered technologies.
