Artificial Intelligence and Mental Health. How did we build our Psychiatric Bot?

[Polska wersja tutaj]

How to automatically recognize mental disorders?

There are many solutions (usually multiple choice forms) for diagnosing mental disorders. They usually focus on one or a few mental disorders, none of them checks for symptoms for every possible mental disorder. For example, there are many questionnaires that assess the intensity of depression or ADHD. There are also a handful of solutions that focus on 2-3 of the most popular disorders at once: for example, ADHD, depression, and anxiety.

However, there are no automatic solutions that simultaneously scan the entire mental health as a psychiatrist or psychologist can do. They only assess the probability of one or several of the many mental disorders, which only constitute a small piece of the puzzle that is mental health. Why hasn't anyone tackled this yet? Well, the questionnaires, on which all other solutions assessing mental health are based, would not pass the test, because:

1. In order to consider every possible mental disorder, such a questionnaire would have to expand to a number of questions impossible for a typical computer user to go through. We attempted to build such a questionnaire here. Despite our best efforts, it ultimately turned out to be over 250 questions.
2. Moreover, mental disorders (although they are mostly classified by symptoms, not causes), rarely have unique concerns. For example, some symptoms of schizophrenia are also attributed to depression, and some to schizoid personality disorder. Of course, this only complicates proper classification.

Despite this, psychiatrists came up with a way to diagnose patients without the need to ask them 250 similar questions, to which most of the answers are "no". What's their secret? They don't use questionnaires, but talk to the patient instead and analyze their open-ended responses.

How to create an algorithm imitating a psychiatrist?

So why hasn't anyone yet built an algorithm that imitates a psychiatrist? Until recently, the automatic analysis of text written by a human was unavailable to us. Fortunately, however, the field of artificial intelligence is developing, and we now have models that can handle Natural Language Processing.

In this article, we present the approach we have taken to create the psychiatrist chatbot, MindMatch. MindMatch chatbot predicts the likelihood of mental disorders and their sources, and matches psychologists, psychotherapists, sexologists, educators, or psychiatrists who have the best experience in this area. This article focuses only on the description of the disorder recognition part, completely omitting therapist recommendations.

Given our modest resources, the project had to be simple to implement. In practice, this eliminated the possibility of training the algorithm. In order to effectively train any text classification model, we would have to gather a huge amount of data, preferably descriptions of patients' mental states classified in terms of mental disorders. Thus, the only available option was to use one of the models known as "zero-shot classification". This term refers to models capable of classifying into categories they did not see during training. Practically, they do not need any additional data, just the text to classify (in our case, what the patient writes to the chatbot) and the categories (in our case, diseases and mental disorders).

However, due to the limitations (described below) of the "zero-shot classification" model, we decided not to classify the text directly into mental illnesses. We broke down this stage into two parts:

1. Classifying text into disorder symptoms using zero-shot classification.
2. "Grouping" symptoms into mental disorders with the help of the XGBoost machine learning model. Thanks to this, we are able to better classify texts into mental disorders.

Text classification model for symptoms

As we explained above, zero-shot classification refers to pre-trained models that have the ability to classify text into unseen classes during training (a nice name for categories). They achieve this by using the similarity of classes to those on which they were originally trained. Therefore, when choosing classes, it was key that they were commonly known words, ideally even seen during the training of the model.

We tested various solutions, starting with clinical names of disorders, then more descriptive aliases for disorders, and finally symptoms grouped into disorders. In the end, we decided to use only very understandable and differentiating symptoms of mental disorders (sadness, delusions, stress, etc.) as labels and then group them into disorders. Thus, we introduced another layer where we collect predictions from the zero-shot classification phase and use a machine learning algorithm to group them into mental disorders.

Mental disorder grouping model based on symptoms

By the term symptom grouping model, we mean a model that analyzes data in the form of a table where columns represent symptoms identified during text classification and values are the probability of these symptoms occurring. This model predicts the likelihood of mental disorders. We applied a standard machine learning algorithm, XGBoost, here. For its training, we used data gathered from patients who had already received a diagnosis, using a special psychological survey. Ultimately, taking into account this additional stage, the whole diagnostic process looks as follows: input data → zero-shot classification identifying symptoms in the text → prediction of diseases based on identified symptoms using XGBoost classification → output data.

Fine-tuning

Once we put the model together, we focused on how to make it better. In the case of zero-shot classification, this meant more precise training of the model and, for XGBoost, parameter optimization. As a result, we fine-tuned the zero-shot classification model using questions that patients asked on psychological forums. To adjust the parameters in the XGBoost algorithm, we used GridSearch, a technique that allows quick checking of the model's effectiveness for a variety of parameter sets. These actions significantly improved the precision of the results generated by our model.

Does it work? Performance and final remarks.

The chatbot generally generates responses that make sense, although it shows a tendency to overpredict borderline personality disorder and rarely predicts ADHD. We decided it's worthwhile to provide users with precise guidelines so that the chatbot receives data that will allow for the extraction of as much valuable information as possible.

1. The text entered by the user should have a minimum of 60 characters but no more than 300. When there is too little information, the algorithm generates results that may seem somewhat random, often overinterpreting the presented symptoms. On the other hand, overly elaborate responses can confuse the chatbot, resulting in the omission of crucial information.
2. We suggest that the user focus on describing their emotions or behaviors and mention what triggers these emotions. The algorithm predicts symptoms, which typically have a strong connection with emotions.

In the future, we plan to expand this tool into a full chatbot, allowing the bot to ask deep-dive questions and adding more questions related to the patient's background (e.g., childhood, parenting experiences, etc.). This will increase the accuracy of the chatbot's predictions as it will be able to "verify" its hypotheses, similar to psychiatrists during diagnosis. We will inform you about any changes on the MindMatch.pl website; we encourage you to follow it. The link to the chatbot is available here (it is in Polish, but you can type in any language).

FAQ - często zadawane pytania

What are the advantages of using AI for mental disorder recognition?

AI offers several advantages for mental disorder recognition, including the ability to analyze large amounts of data quickly and objectively. It can identify subtle patterns and changes in behavior that might be missed by human clinicians. AI can also provide consistent assessments, reducing the risk of human bias. Moreover, it has the potential to reach a wider audience, offering mental health support to individuals who may not have access to traditional healthcare services.

What are the challenges and limitations of AI in recognizing mental disorders?

Despite its potential, AI in mental disorder recognition faces challenges and limitations. One major challenge is the need for high-quality and diverse training data, as well as ensuring privacy and data security. AI models may not always capture the complexity of human emotions and behaviors accurately. Additionally, AI should be considered a supportive tool for healthcare professionals rather than a replacement, as it lacks the empathy and human touch needed for comprehensive mental health assessments. Ethical concerns, such as potential misuse or bias in AI algorithms, also need to be carefully addressed in the development and deployment of such technology.

Can AI assist in early intervention and prevention of mental health disorders?

Yes, AI can play a significant role in early intervention and prevention. By continuously monitoring individuals for changes in behavior, AI systems can alert healthcare professionals or caregivers to potential issues before they escalate. This proactive approach allows for timely intervention, reducing the severity of mental health conditions and improving treatment outcomes. Additionally, AI-driven educational campaigns and mental health resources can help raise awareness and provide tools for managing stress and emotional well-being, contributing to prevention efforts.