Moderate evidence
More
Turkish Airlines chatbot study finds exact-match gains, but less accurate routing
Study title: Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot
- Evidence type
- Controlled or validated comparison
- Sample
- n = 80,295
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-3097cite or check it
Study title: Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot
Researchers analysed 80,295 Turkish WhatsApp utterances from Turkish Airlines' TK Assistant chatbot, collected from January to July 2025. After a Pareto analysis, they added 349 high-scoring utterances that had no verbatim counterpart to ten sub-intent libraries.
Read the study abstract
This study asks whether the Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) cycle can improve utterance capture in a live enterprise chatbot, and how much of the change is attributable to the intervention during a concurrent migration to large language model (LLM) components. Turkish WhatsApp utterances of Turkish Airlines’ TK Assistant (80,295 utterances, January–July 2025) was analyzed; utterance capture scores (success threshold 0.95) were graded on a five-level scale, with 0.95 as the success th…
The gain
In a field study of Turkish Airlines' TK Assistant WhatsApp chatbot, 17 of 22 recurring added texts were captured as exact matches at first recurrence, versus none of eight recurring non-added controls.
In brief
Added texts that came back were captured as exact matches far more often: 17 of 22 at first recurrence, against none of eight recurring controls, a 77-point difference. The effect did not spread beyond near-identical forms of the added text.
Overall, the successful utterance capture rate fell from 21.04% to 16.11% over the period. The authors say that decline cannot be separated from the chatbot's concurrent migration to large language model components. Blind annotators judged 87% of exact matches correctly routed versus 98% of Strong-band matches, leading the authors to conclude that verbatim curation buys exact-match efficiency, not understanding.
Main points
- The dataset was 80,295 Turkish WhatsApp utterances from Turkish Airlines' TK Assistant between January and July 2025, scored against a 0.95 success threshold.
- 349 Strong-band utterances without a verbatim counterpart were added to ten sub-intent libraries after a bidirectional Pareto analysis.
- 17 of 22 recurring added texts became exact matches at first recurrence, versus none of eight recurring controls (difference 77 points, 95% confidence interval 39–90).
- Added texts were not promoted organically beyond near-identical forms.
- Under stated assumptions, the intervention was followed by 69 additional exact matches, while the successful capture rate fell from 21.04% to 16.11%.
- In a blind validation of 300 utterances, exact matches were judged correctly routed 87% of the time versus 98% for Strong-band matches.
The full story
The team applied the Six Sigma DMAIC cycle to utterance capture in a live enterprise chatbot. Assessment combined utterance-level re-capture analysis against non-added controls, expected-versus-actual attribution, Laney p′ charts and blind human validation.
The curation worked narrowly: recurring added texts were matched exactly, but the library did not generalise to similar phrasings. Under the stated assumptions, 69 additional exact matches followed the intervention.
The headline capture rate moved the other way, falling from 21.04% to 16.11%, during a concurrent migration to large language model components. The authors say confidence-based metrics should be secondary improvement targets, since exact-match gains did not mean better routing.
Sources
1 on file- Applied Sciencesdoi:10.3390/app162211506
Talk it over
Who gets this benefit, and who was left out of the study or story?
Turkish Airlines chatbot study finds exact-match gains, but less accurate routinghttps://truvace.com/item/efficiency-without-quality-a-six-sigma-dmaic-define-measure-analyze-improve-cont
Who gets this benefit, and who was left out of the study or story?
The gain In a field study of Turkish Airlines' TK Assistant WhatsApp chatbot, 17 of 22 recurring added texts were captured as exact matches at first recurrence, versus none of eight recurring non-added controls.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a controlled comparison with a sample of 80,295, from Applied Sciences.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat The drop in successful capture rate from 21.04% to 16.11% cannot be separated from the concurrent migration to large language model components, so the intervention's effect on overall performance is uncertain.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
ace



What readers say