Evaluation of an expert system for the generation of speech and language therapy plans

Background Speech and Language Pathologists (SLPs) have to deal with a wide spectrum of disorders, which may arise from many different conditions and affect voice, speech, language and swallowing capabilities in disparate ways. Therefore, the outcomes of Speech and Language Therapy (SLT) are highly dependent on the accurate, consistent and complete design of personalized therapy plans. However, SLPs often have very limited time to work with their patients and to browse the large (and growing) catalog of activities and specific exercises that they may put into the therapy plans. In consequence, many plans are suboptimal and fail to address the specific needs of each patient. Objective This paper evaluates an expert system whose aim is to automatically generate plans for speech and language therapy, containing semiannual activities in the five areas of hearing, oral structure & function, linguistic formulation, expressive language & articulation, and receptive language. The goal is to assess whether the expert system can speed up the SLPs’ work and lead to more accurate, consistent and complete therapy plans for their patients. Methods This paper presents the evaluation results of the SPELTA expert system in supporting the decision making of 4 SLPs who are treating children in 3 institutions of special education in Ecuador. The expert system was first trained with data from 117 cases, including medical data, diagnosis for voice, speech, language and swallowing capabilities, and therapy plans created manually by the SLPs. Then, it was used to automatically generate new therapy plans for 13 new patients. The SLPs were finally asked to evaluate the accuracy, consistency and completeness of those plans. A 4-fold cross-validation experiment was also run on the original corpus of 117 cases in order to assess the significance of the results. Results The evaluation showed that 87% of the outputs provided by the SPELTA expert system were considered valid therapy plans for the different areas. The SLPs rated the overall accuracy, consistency and completeness of the proposed activities with 4.65, 4.6 and 4.6 points (to a maximum of 5), respectively. The ratings for the sub-plans generated for the areas of hearing, oral structure & function and linguistic formulation were nearly perfect, whereas the sub-plans for expressive language & articulation and receptive language failed to deal properly with some of the subject cases. Overall, the SLPs indicated that 92% of the sub-plans generated automatically were “better than” or “as good as” what the SLPs would have created manually if given the average time they can devote to the task. The cross-validation experiment yielded very similar results. Conclusions The results show that the SPELTA expert system provides valuable input for SLPs to design proper therapy plans for their patients, in much shorter time and considering a larger set of activities than proceeding manually. The algorithms worked well even in the presence of a sparse corpus, and the evidence suggests that the system will become more reliable as it is trained with more subjects.