Best Practices for A/B Testing Machine Learning Models (2023)

By Piper Williams, Charter Communications; Ryan Lewis, Charter Communications; Miranda Kroehl, Charter Communications; Veronica Bloom, Charter Communications; Brock Bose, Charter Communications

Customization of support sites across self-service platforms can significantly enhance overall customer experience and satisfaction. At Charter Communications, one of our primary objectives in this area is to drive improvements to digital troubleshoot and support pages by optimizing digital support content, content “findability” and content rules. By customizing support content, we aim to reduce the time between our customers’ questions and their resolutions, ease the navigation process for them by surfacing relevant content and eliminate the need for a customer service phone call. Machine learning offers powerful capabilities to customize content by leveraging vast amounts of data across several areas to predict support-seeking behaviors. These predictions enable us to proactively surface the right support articles pertinent to users at the right time. It’s not enough to build these machine learning models, which is a large feat in and of itself. Once a model is built and validated, monitoring model performance over time is necessary to make sure the models are performing as expected. Traditional model performance metrics, such as precision, recall and area under the receiver operating characteristic curve (AUC-ROC) allow data science teams to assess model accuracy and prevent model drift. In addition, a successful machine learning application also has a positive and significant impact on key performance indicators (KPIs), which requires experimentation (i.e., A/B testing) and robust statistical analyses. To assess the full impact of machine learning-driven experiences, it’s critical to implement a systematic process that rigorously and iteratively tests, validates and optimizes these applications. At Charter Communications, our Data Science teams have developed the Experimentation-Machine Learning Lifecycle Process, a standardized set of best practices for operationalizing machine-learning driven content rules. This process consists of five phases, which include 1) Model Discovery A/B, 2) Productionalization of the Model, 3) Model Implementation A/B, 4) Operationalization of the Machine Learning Application and 5) Continuous Improvement. In this paper, we walk through the Experimentation-Machine Learning Lifecycle Process, describe how to carry out each of the five phases for any machine learning application and provide examples for how these phases apply to our own teams’ work. In doing so, we provide a set of guidelines that can be implemented by others to evaluate and optimize their machine learning applications.

By clicking the "Download Paper" button, you are agreeing to our terms and conditions.

Similar Papers

New Generation Data Governance for Charter Network:1
By Jay Liew, Mark Teflian, Bruce Bacon, Jay Brophy & Randy Pettus, Charter Communications
Composite Quality Metric KPIs - A General Framework for Interpretable Composite Metrics
By Rohan Khatavkar, Ryan Lewis, Veronica Bloom; Charter Communications
Scaling IP Advertising Using Manifest Manipulation
By Vipul Patel, Charter Communications; Xavier Denis, CommScope
From Support Calls to Insights - Using Automatic Speech Recognition and Natural Language Processing to Drive Product Roadmaps
By Jing Qing, Veronica Bloom, Michael Addonisio, Christy Gearheart; Charter Communications
On The Performance Of CBRS Fixed Wireless Access: Coverage And Capacity Field Study
By Mohamed Daoud, Matthew Hubbard, Rajeev Aggarwal & Hossam Hmimy, Charter Communications
Communications For Improved Quality Of Life
By Dr. Peter C. Goldmark, Goldmark Communications Corp.
Operational Transformation Via Machine Learning
By Shamil Assylbekov, PhD. & Devin Levy, Charter Communications
Hybrid Cloud to Power Private 5G Networks: Industrial Worker Safety Use Case
By Mohamed Daoud, Charter Communications; Ammar Latif, Amazon Web Services (AWS)
Smart Building Internet of Things Deployment Using Low Power Wide Area Networks
By Victor Lopez, Charter Communications; Mohamed Daoud, Charter Communications; Hossam Hmimy, Charter Communications
Hybrid Fiber Coaxial (HFC) Spectrum Efficiency and Quality - Systematically Evolving Networks using Profile Management Platform (PMA)
By Jay Liew, Charter Communications
More Results >>