Automated email answering by text-pattern matching: Performance and error analysis

Automated answering of frequent email inquiries can be implemented as a text categorization task with narrow text categories, where all messages in 1 text category have the same answer. Such email categorization should be optimized for high precision and at least acceptable recall. One such high-precision email categorization method is matching of surface text patterns to incoming email messages. In order to assess the upper performance limits of text-pattern matching, we conducted extensive tests with almost 10,000 messages. Our results show that automated email answering with precision around 90% and recall 50–75% is feasible. In order to achieve this, however, the system must work with multiword expressions rather than stand-alone words. Furthermore, we argue that the system has to distinguish the context of an email inquiry from the actual need that created the inquiry—a question, request, or complaint. We have discovered and analysed 12 reasons why text-pattern matching may fail.