Multimodal Data Fusion for Depression Detection on Twitter

Document Type : Original Article

Authors

Computer Science Department, Faculty of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran

Abstract

Depression is one of the most significant and prevalent mental health disorders in today’s world. Early detection of depression is critical, and this study aims to identify depression in individuals using information derived from social media. The use of social media for various purposes has grown in recent years, as these platforms provide valuable insights into both individuals and society. Social media can be effectively utilized to detect depression. Researchers have attempted to identify depression using various types of data, such as images, text, and audio. Most studies have focused on using only one type of data, such as text or images, for detection. While these methods have achieved notable results, they have limitations in accuracy that can be addressed by incorporating new methods and integrating multiple data modalities into the model. In this study, we propose a multimodal model that analyzes text and images together to detect depression. Compared to similar models, our approach achieves an approximate 5% improvement in accuracy, reaching 89.87%, while utilizing significantly less of the original dataset.

Keywords

Main Subjects

References

[1] A. Jitimon, T. Suda, and A. Thara, "Big data analytics on social networks for real-time depression detection," *Journal of Big Data*, vol. 9, p. 69, 2022.
 
[2] G. Shelley, S. Archana, and R. Jayanthi, "Multimodal, multiview and multitasking depression detection framework endorsed with auxiliary sentiment polarity and emotion detection," Jan. 19, 2023.
 
[3] C. B. Krishna, T. T. Marijn, L.-L. Lorenzo, A. R. Lauren, and B. Johan, "Individuals with depression express more distorted thinking on social media," *Nature Human Behaviour*, vol. 5, pp. 458–466, 2021.
 
[4] S. C. Gullal, H. Sherzod, S. Abdul, M.-B. Eric, O. Christian, and R. E., "MM-Claims: A Dataset for Multimodal Claim Detection in Social Media," *arXiv preprint arXiv:2205.01989*, 2022.
 
[5] B. Marouane, K. Mohammed, and B.-H. Abderrahim, "A comprehensive survey on sentiment analysis: Approaches, challenges and trends," 2023.
 
[6] F. Cummins, D. Doyle, and G. McCarthy, "Towards Automatic Text-Based Estimation of Depression Through Symptom Prediction," 2022. [Online]. Available: https://example-link1.com
 
[7] A. Ibitoye, O. Bamidele, and K. Dahunsi, "Deep Learning for Depression Detection from Textual Data," 2021. [Online]. Available: https://example-link2.com
 
[8] D. L. Danxia, F. X. Lin, A. Farooq, S. Muhammad, and G. Jing, "Detecting and measuring depression on social media using a machine learning approach: Systematic review," *JMIR Mental Health*, vol. 9, p. e27244, 2022.
 
[9] Z. Li, A. Zhengyi, C. Wenchuan, Z. Jiawei, Z. Fang, and H. Bin, "MHA: A multimodal hierarchical attention model for depression detection in social media," *Health Information Science and Systems*, vol. 11, p. 6, 2023.
 
[10] M. A. H. Yazda, M. Saeid, B. Goonmeet, R. William, S. Amit, A. Hassan, T. Krishnaprasad, M. John, M. Annie, and H. Pascal, "Multimodal mental health analysis in social media," *PLoS One*, vol. 15, p. e0226248, 2020.
 
[11] A.-M. Bucur, A. Cosma, P. Rosso, and L. P. Dinu, "It's just a matter of time: Detecting depression with time-enriched multimodal transformers," in *European Conference on Information Retrieval*, Springer, Jan. 13, 2023, pp. 200–215.
 
[12] O. Adekanmbi, M. A. Adekanmbi, M. O. Adekanmbi, and O. O. Adekanmbi, "Algorithmic teenagers’ depression detection on social media and automated instant engagement using therapy bot powered by multimodal deep learning and psychotherapy intervention," 2023.
 
[13] G. Shen, J. Jia, L. Nie, F. Feng, C. Zhang, T. Hu, T.-S. Chua, and W. Zhu, "Depression detection via harvesting social media: A multimodal dictionary learning solution," *IJCAI*, pp. 3838–3844, 2017.
 
[14] T. Gui, L. Zhu, Q. Zhang, M. Peng, X. Zhou, K. Ding, and Z. Chen, "Cooperative multimodal approach to depression detection in Twitter," in *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 33, no. 01, pp. 110–117, 2019.
 
[15] P. Xu, X. Zhu, and D. A. Clifton, "Multimodal learning with transformers: A survey," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 10, pp. 12113–12132, 2023.
 
[16] A.-M. Bucur, A. Cosma, P. Rosso, and L. P. Dinu, "It's just a matter of time: Detecting depression with time-enriched multimodal transformers," in *European Conference on Information Retrieval*, Springer, pp. 200–215, 2023.
 
[17] P. Mann, A. Paes, and E. H. Matsushima, "See and read: detecting depression symptoms in higher education students using multimodal social media data," in *Proceedings of the International AAAI Conference on Web and Social Media*, vol. 14, pp. 440–451, 2020.
 
[18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in *Proceedings of NAACL-HLT*, pp. 4171–4186, 2019.
 
[19] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 770–778, 2016.
 
[20] H. Liu, Y. You, and D. Darrow, "Deep Incubation: Training Large Models by Divide-and-Conquering," *arXiv preprint arXiv:2212.04129*, 2022.
 
[21] J. Wang, Z. Liu, and T. Zhang, "Dataset Pruning: Reducing Training Data by Examining Generalization Influence," *arXiv preprint arXiv:2205.09329*, 2022.
 
[22] T. F. Qu, Y. Zhang, and J. Liu, “Depression Detection Using Emotion Artificial Intelligence,” in 2019 IEEE 10th International Conference on Awareness Science and Technology (iCAST), Morioka, Japan, 2019, pp. 1-5.
 
[23] S. Orabi, P. Buddhitha, M. H. Orabi, and D. Inkpen, “Deep Learning for Depression Detection of Twitter Users,” in Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, New Orleans, LA, USA, 2018, pp. 88-97.
 
[24] F. Ren and Y. Wu, “Predicting Depression Trends on Twitter Using Sentiment Analysis,” in 2019 IEEE 13th International Conference on Semantic Computing (ICSC), Newport Beach, CA, USA, 2019, pp. 230-233.
 
Applied and basic Machine intelligence research
Volume 2, Issue 1
September 2024
Pages 120-130

Files

Share

How to cite

Statistics