A mind monitoring system: Voice analysis technology

Shinichi Tokuno from Voice Analysis and Measurement of Pathophysiology, Department of Bioengineering, Graduate School of Engineering, at the University of Tokyo, provide an overview of “MIMOSYS”, a “mind monitoring” system that uses voice analysis technology

In the October 2019 edition of this journal, we discussed the potential of voice biomarkers. Here, we will provide an overview of “MIMOSYS”, a “mind monitoring” system mentioned in that edition.

“MIMOSYS”, which stands for Mind Monitoring System, is a smartphone application that automatically monitors the mental health of users by deconstructing and analysing their voices during phone calls. In humans, emotional expressions are blunted by stress and depression. However, our system monitors the mental health of users based on the relative abundance of different emotions in their voices. The system integrates AGI Inc.’s emotion recognition software, “Sensibility Technology” (ST), to identify and compare the levels of four emotions – joy, anger, sorrow, and calmness – along with excitement, based on prosodic features.

The technology is already employed in the entertainment sector in Japan, via portable games and smartphone apps and in call centres and other industrial settings. We have confirmed that there is a good correspondence between the emotional activities identified by ST and the activation of brain regions involved in the emotional processes during functional magnetic resonance imaging (fMRI) (Mitsuyoshi et al., 2011). We have also confirmed that such activities change in response to stress (Tokuno et al., 2011).

What was the outcome of the research?

These findings encouraged us to develop an index to detect emotional changes, for monitoring the mental health of users (Mitsuyoshi et al., 2017). The prototype “psycho-analyser” software developed to this end was unaffected by reporting bias, which is the biggest limitation of self-assessments for stress screening and had comparable sensitivity to questionnaires such as the General Health Questionnaire-30 items (GHQ-30) (Tokuno, 2018). However, gender differences were apparent in the emotional scores generated by the “psycho-analyser” software.

“Joy” tended to be higher among women and “sorrow” among men. To eliminate these differences, the proxy index of “vivacity” was derived from “joy” and “sorrow”, and the “relaxation” index was derived from the emotions of “calm” and “excitement.” Finally, “vitality” was derived from “vivacity” and “relaxation” as a composite index of mental health (Tokuno, 2018). However, the “vitality” index suffered from low specificity, because it was highly dependent on the content and participants in the conversations of the users.

This challenge was overcome by developing an additional index, “mental activity”, from the two-week mean and variation of the “vitality” index (in press).

We have validated the “vitality” and “mental activity” – indices and examined their benefits in over 10,000 subjects across a wide range of situations. For example, we have demonstrated that the “vitality” index is viable in efficacy assessments of stress interventions (Shinohara et al., 2015) and for correlation with depression severity measured by the Beck Depression Inventory (BDI) when administered under conditions of minimal reporting bias (Hagiwara et al., 2017). Additionally, we have demonstrated that the “vitality” index performs equally well for voice recordings both before and after the processing of call transmissions (Hagiwara et al., 2017 and Omiya et al., 2018).

What is the best use for “MIMOSYS”?

Such pieces of evidence support the application of the “vitality” index in the assessment of the efficacy of screenings and interventions. However, the greatest advantage of “MIMOSYS” is its ability to monitor the mental health of users based on the “mental activity” index. Our technology has demonstrated expected results in the early detection of excessive stress among employees in several workplace environments (Miyazaki et al., 2016; see figure 1). We continue to refine and utilise “MIMOSYS”, which is now – a public research tool, that has been downloaded nearly 6,000 times to date by users around the world. We also continue to search for connections between index scores and various blood biomarkers.

Our tests have led to the commercialisation of “MIMOSYS” as a support tool for occupational mental health. Furthermore, individuals are benefited from smartphone applications that are integrated with our algorithms and now come preinstalled on multiple devices. This has yielded a de facto user base of over 1.5 million. Our efforts in the future will focus on popularising the technology both domestically through collaboration with local governments and globally by translating and validating the software in multiple languages.

Subjects with low scores for the “mental activity” index recovered after consulting an industrial physician. A later survey revealed that these individuals had been struggling with matters of concern in their private lives. Figure 1 is based on the material presented at the 7th Asia Pacific Regional Conference of the International Association for Suicide Prevention.

References

Mitsuyoshi et al., 2011, DOI: 10.1109/DSR.2011.6026845
Tokuno et al., 2011, DOI: 10.1109/DSR.2011.6026823
Mitsuyoshi et al., 2017, DOI: 10.1186/s40696-017-0032-0
Tokuno, 2018, DOI: 10.1007/978-981-10-6577-4_6
Shinohara et al., 2015, DOI: 10.1007/978-3-319-32270-4_5
Hagiwara et al., 2017, DOI: 10.25046/aj020343
Hagiwara et al., 2017, DOI: 10.1109/EMBC.2017.8036807
Omiya et al., 2018, DOI: 10.25046/aj030555
Miyazaki et al., Kokoro, and Syakai, 2016,　47(3), 26-31 (Japanese).

*Please note: This is a commercial profile