Unveiling the Value of Chatbot Testing
“What is the value of testing chatbots?” — A question that hits right in the heart of every passionate tester. Seeing it again in my inbox, it’s time to shed some light on this topic.
First - Chatbots are Software
“Thank you, Captain Obvious” is what you might think. But in fact, that’s not that clear for everyone. Conversational AI, meaning chatbots and virtual assistants are Software programmes. And Software needs to be tested. This is common sense in the industry right now but it took us decades of time and billions of error fixing costs in production to learn it.
According to the National Institute of Standards and Technology (NIST) Software bugs cost the U.S. economy $60 billion every year.
Second - The value of testing
The value of testing is derived from quality. And quality means the satisfaction a user can feel every time he talks to a chatbot or virtual assistant. This is why we call the whole discipline “Quality Assurance”. And of course confidence. A high confidence level to ship products into production environments is from utmost importance in agile projects.
Automation is the key enabler. Test automation leads to early identification of defects and unwanted behaviour. Decreasing the time to market as a result and the costs of quality while amplifying the end-user satisfaction level at the same time.
Third - The holistic test approach of Botium
Everything here at Botium started with an open source test automation framework for bots called Botium Core. This pretty soon became the industry standard for testing conversation flows but in fact it was missing some key features. Therefore we introduced Botium Box to offer a holistic test approach for chatbots.
The following image represents the BDLC (Bot Development Life Cycle). Milestones in every phase are shown underneath the gray line. Starting from the left with analyzing the user needs, deciding for your bot channels, followed by the NLP selection and so on. Above the gray line, you can see the holistic approach of Botium by adding value in every phase.
PLAN: Data sets for different domains help to identify actual user needs. Provider benchmark tests done with Botium Coach are the foundation to choose the right NLP engine for your bot.
DESIGN: Based on the conversation flow designed, Botium Box will automatically generate data sets for testing and training in just a few seconds. This a very good starting point to automate your chatbot.
DEVELOP: The Botium Crawler will crawl through the entire conversation tree of your chatbot and generate data sets based on the data collected. And if you add speech support to your bot, Botium has built in voice based testing.
TRAIN: The natural language processing and it’s NLP score are continuously challenged using Botium Coach.
TEST: Conversation flow testing, NLP score testing, real End-to-End testing, voice based testing, testing IVR systems, performance testing, security and GDPR testing just to name a few superpowers.
DEPLOY: Contantly keep an eye on your chatbot in production using Botium Monitoring.
Wrap-Up
Chatbots are Software and Software needs to be tested. The value is dereived from quality and the end-user satisfaction. To keep both at the highest level possible, Botium Box offers a holistic test approach for conversational AI with many superpowers.
