Fix It With Bluejay
Hi, I'm Azfar, and I'll be taking over the newsletter this week!
I joined Bluejay as a storyteller just days ago, and the first thing I did was throw myself into actually building my first voice agent. No better way to learn, right?
Here's exactly how I did it, what broke, how I fixed it, and what you should do differently if you're building your first voice agent.
From Idea to Agent in Coffee Shop
Over the last two years, I've watched coffee shops explode all over the San Francisco Bay Area. Always new spots opening everywhere, lines out the door, oat milk everything. But every time I spoke to an owner, the conversation came back to the same thing.
"People keep calling us asking if we take reservations."
The answer is always no. Coffee shops run on walk-ins. But the calls kept coming, tying up staff and pulling attention away from customers already in the room. Same question. Every single day. So I thought, why not build an agent that just handles that?
I named him Oswald.
I created Oswald with an agent builder platform, and used OpenAI, Deepgram, and ElevenLabs to build a cascaded model architecture. Then, I wrote him a full system prompt covering everything – the walk-in only policy, exact opening hours, the menu, and how to handle catering inquiries. Once he was set up I ran a quick test just to make sure he was alive. Sure enough, he was.
The real test came when I brought him into Bluejay. To make this process seamless I used Bluejay AI, our in-app co-pilot capable of doing anything on the platform, from running a simulation to analyzing results and recommending fixes to my agent’s system prompt (more on this in the Feature Spotlight below!).
I kicked off a simulation, created three digital humans to cover different caller scenarios, and let the simulation run.
Oswald's first simulation in Bluejay
Bluejay found 4 different failures in my first run. I was glad that I ran these simulations first. If a customer experienced these failures, Oswald would be directly responsible for losing orders!
Bluejay broke down every single failure and the issues were way more specific than I expected:
- Oswald was answering questions about hours but never giving exact times, just vague ballpark answers that didn't meet the success criteria.
- When callers asked about catering he'd acknowledge it but never fully commit to collecting the right information, things like full name, company, guest count, preferred date, and email.
- He also forgot to mention the 24 to 48 hour follow up window entirely.
- In one run he referred to Qishr, a traditional Yemeni spiced coffee husk drink, in a way that made it sound like Quiche - which is a type of pastry.
Bluejay AI flagging the issues
Generally speaking, Bluejay showed me that my prompt was missing key sentences that could make it better, and it recommended fixes that I could copy paste into my agent to fix the issues it surfaced.
So I went to Bluejay AI and asked it directly:
"What prompt can I use to fix this inside my agent?"
It came back with something tighter and more structured that addressed every single failure point.
- State exact hours.
- Treat catering as a mandatory lead capture flow.
- Define Qishr properly.
- Commit to the right workflow the moment the caller's intent is clear.
I copied it, jumped back into my agent, pasted it in, and hit save.
Here's the exact prompt Bluejay AI gave me:
Prompt Fix
Intent routing rule:
As soon as you understand the caller's intent, switch into the matching workflow and complete it fully. Do not stay generic once the intent is clear.
Catering and wholesale workflow:
If a caller mentions catering, wholesale, a company event, business order, private event, or large gathering, you must immediately switch into lead-capture mode.
First respond with enthusiasm, for example:
"Absolutely — we'd love to bring Yemeni coffee culture to your event."
Then collect these details one at a time:
- full name
- company or event name
- estimated guest count or order size
- preferred date
- email address
After collecting the details, say:
"Our team will follow up within 24 to 48 hours with a custom quote."
Rules for catering and wholesale:
- Do not give pricing
- Do not skip the information collection steps
- Do not end the call before attempting to collect all required details
- Treat catering as a lead-capture workflow, not a general FAQ
Menu and drinks rule:
Qamaria serves drinks only: coffee and tea.
There is no food menu.
Never invent or imply food items.
Qishr clarification rule:
Qishr is a drink, not a food item.
Always describe it as a traditional Yemeni spiced coffee husk drink.
Never say or imply "quiche."
If helpful for clarity, say:
"Qishr — spelled Q-I-S-H-R — is one of our signature Yemeni drinks."
Menu response rule:
If a caller asks what is on the menu, clearly say:
- Qamaria serves coffee and tea drinks only
- there is no food menu
- signature offerings include Qishr, single-origin Yemeni coffee, and traditional Yemeni teas
- Yemen is considered the birthplace of coffee
Do not mention any item unless you are sure it is part of the drinks menu.
Oswald's second simulation
Those same scenarios. The same digital humans. Every single test passed. The hours and location, catering inquiry, menu and heritage. All of it had simulated very well !
I never built a voice agent before in my life, and I went from a problem I spotted in a coffee shop to a fully tested, broken, fixed, and functioning agent in a single day. The process taught me one thing above everything else. Your agent is only as good as the prompt and context behind it, and you will not know what is wrong with that prompt until you actually test it.
Bluejay AI will find what you miss. And then it will tell you exactly how to fix it !
As a recap:
- Bluejay is a testing and monitoring platform for Conversational AI agents. Companies ranging from Fortune 10 enterprises to fast-growth startups in the Silicon Valley use Bluejay to make sure their voice and text agents work in production (monitoring) and development (testing) environments.
- Our team, now seven strong, works around the clock to make sure your agent behaves when talking to customers.
- This newsletter is 100% human written. It always has been, and it always will be. Ask yourself about what you are consuming. If the writer hasn't read it, why should you?
The Voice AI community is growing, and we're so happy to facilitate its growth here at Bluejay. After all, it plays directly into our mission:
Announcements
Heres what happened at Bluejay last week:
- Bluejay attended the Twilio Signal and Cerebral Valley Voice conferences this week, where Rohan met exciting builders and leaders in Voice AI!
- The latest Skywatch episode just dropped featuring AssemblyAI, marking Azfar's first piece of work for Bluejay as our storyteller.
- The team is growing, with two new hires joining this week.
- The team pushed 38,833 lines of code this week to make Conversational AI more reliable.
Skywatch Podcast Episode
From PHP Books to Neural Networks: Assembly AI CEO Dylan Fox on the Evolution of Voice AI
This week Rohan sat down with Dylan Fox, Founder and CEO of AssemblyAI, to talk about how voice infrastructure got broken and how Dylan set out to fix it.
From getting mailed a CD-ROM just to try a voice API, to building one of the most developer-friendly speech platforms out there, the conversation covers what most teams get wrong when building with voice.
Full episode on YouTube and Spotify now. 😄
Feature Spotlight: Bluejay AI ! Our Internal Copilot
Improve your agent's issues and validate improvements with Bluejay AI
Is your agent having production issues and you need to validate those improvements ? With Bluejay AI you now have a copilot which can improve your voice and chat agents.
The crazy thing is it can analyze production calls, find those failure patterns, generate simulations, explain why those tests failed, and my favorite is when it suggests prompt changes you can make based on the failures that you typically find in your production and simulated calls. At the end you can rerun the evaluation to verify those improvements.
That's all for now. I'll see you next time!
Azfar Khan
Storyteller @ Bluejay
