CC BY
1
_МЕДИЦИНСКАЯ ИНФОРМАТИКА / MEDICAL INFORMATICS_
DOI: https://doi.org/10.23670/IRJ.2023.133.40
DEVELOPMENT OF SYSTEMS FOR REAL-TIME RECOGNITION OF HUMAN PSYCHOPHYSIOLOGICAL STATE AS PART OF THE GROWTH OF INFORMATION HEALTH-PRESERVING TECHNOLOGIES
Research article
Ilichev V.Y.1' *, Lavrentyeva G.V.2, Gerasimova N.S.3, Kashirin D.S.4
1 ORCID : 0000-0003-1017-5544;
2 ORCID : 0000-0003-2962-8427;
3 ORCID : 0000-0002-6114-7669;
4 ORCID : 0000-0001-8942-1925;
1 2' 3' 4 Kaluga Branch of Bauman Moscow State Technical University, Kaluga, Russian Federation
* Corresponding author (patrol8[at]yandex.ru)
Abstract
The ability to manage emotions, notice them in time, and also cope with them can serve as an incentive for human development. On the contrary, the inability to recognize emotions and control them can cause long-term stress, as well as have a negative impact on human life and health, and cause cancer.
The purpose of the work is to create an algorithm for developing a system for automated recognition of the psychophysiological state of a person in real time by analyzing biological signals (electroencephalogram (EEG), electrocardiogram (ECG), photopletismogram and body temperature) from the side of physiological human health. In the course of the study, it is assumed to measure the emotional and physiological responses of an individual using wearable devices in various conditions, highlight significant changes in biological signals during emotional states, recognize standard patterns for each emotional response, use a fast Fourier transform to extract spectral information from signals, as well as artificial neural networks (ANN) as an adaptive method for solving the problem.
The developed technology will help to cope with the problem of "emotional burnout", better understand people and their behavior, and develop intelligence. The results of the study can be used for medical purposes as an assistant for making a correct diagnosis to the patient, and the algorithms tested by the experiment will serve as the basis for creating a mobile system for automatic assessment and monitoring of the psycho-emotional state of the individual.
Keywords: electrocardiogram, electroencephalogram, smartwatch, fast Fourier transform, neural network, Python language.
РАЗРАБОТКА СИСТЕМ ПО РАСПОЗНАВАНИЮ ПСИХОФИЗИОЛОГИЧЕСКОГО СОСТОЯНИЯ ЧЕЛОВЕКА В РЕЖИМЕ РЕАЛЬНОГО ВРЕМЕНИ В РАМКАХ РАЗВИТИЯ ИНФОРМАЦИОННЫХ
ЗДОРОВЬЕСБЕРЕГАЮЩИХ ТЕХНОЛОГИЙ
Научная статья
Ильичев В.Ю.1' *, Лаврентьева Г.В.2, Герасимова Н.С.3, Каширин Д.С.4
1 ORCID : 0000-0003-1017-5544;
2 ORCID : 0000-0003-2962-8427;
3 ORCID : 0000-0002-6114-7669;
4 ORCID : 0000-0001-8942-1925;
i, 2, 3, 4 Калужский филиал Московского государственного технического университета имени Н.Э. Баумана, Калуга,
Российская Федерация
* Корреспондирующий автор (patrol8[at]yandex.ru)
Аннотация
Умение управлять эмоциями, вовремя замечать их, а также справляться с ними может служить стимулом к развитию человека. Напротив, неумение распознавать эмоции и контролировать их может вызывать долговременный стресс, а также оказать негативное влияние на жизнь и здоровье человека, стать причиной онкологических заболеваний.
Целью работы является создание алгоритма разработки системы автоматизированного распознавания психофизиологического состояния человека в режиме реального времени путём анализа биологических сигналов (электроэнцефалограммы (ЭЭГ), электрокардиограммы (ЭКГ), фотоплетизмограммы и температуры тела) со стороны физиологического здоровья человека. В ходе исследования предполагается измерение эмоциональных и физиологических реакций индивида при помощи носимых устройств в различных условиях, выделение значительных изменений в биологических сигналах во время эмоциональных состояний, распознавание стандартных паттернов для каждой эмоциональной реакции, использование быстрого преобразования Фурье для извлечения спектральной информации из сигналов, а также искусственных нейронных сетей (ИНС) в качестве адаптивного метода для решения поставленной задачи.
Разработанная технология поможет справляться с проблемой «эмоционального выгорания», лучше понимать людей и их поведение, развивать интеллект. Результаты исследования могут быть использованы в медицинских целях в качестве помощника для постановки правильного диагноза пациенту, а проверенные экспериментом алгоритмы
послужат основой для создания мобильной системы автоматической оценки и мониторинга психоэмоционального состояния индивида.
Ключевые слова: электрокардиограмма, электроэнцефалограмма, умные часы, быстрое преобразование Фурье, нейросеть, язык Python.
Introduction
The acceleration of the rhythm of life and the exponential increase in information noise causes severe stress loads acting on a modern person. To cope with so much information coming from outside, the human brain is increasingly forced to resort to methods of saving internal resources: artificially put barriers to incoming information, resist learning, automate responses to stimuli. This, in turn, provokes people to unconscious actions, causes apathy, leads to depression and interferes with the development of the student. In this regard, the problem of the development of the emotional intelligence of an individual is acute. Increasingly, a person wonders: "How to maintain your health and the health of others in the conditions of accumulated stress?" The technology of automated recognition of a person's psychophysiological state will help to cope with the problem, as well as better understand people and their behavior.
Emotions are a person's psychophysiological or mental responses to objects and events of the world around them. Emotions affect various aspects of human life, including attention, learning, development, goal achievement, communication, mood, and motivation [1].
The ability to manage and cope with emotions can be an incentive for human development and a source of energy. On the contrary, inability to recognize emotions and control them can cause long-term stress, as well as have a negative impact on human life and health, be the cause of cancer [2]. According to experts, emotions themselves are neutral. Our reaction gives them color. It depends on it how the emotion will affect our well-being and health [3].
With the development of technologies and wearable gadgets (smartwatches and other portable devices), with the improvement of sensors built into them or connected to them to record changes in the electroencephalogram (EEG), electrocardiogram (ECG), photopletismogram and body temperature, the most convenient way to recognize emotions in real time is to analyze biological signals from the human body [4].
The purpose of the study is to create an algorithm for developing a system for automated recognition of a person's psychophysiological state in real time by analyzing biological signals. This algorithm is supposed to be tested by conducting a number of experimental studies on volunteers.
Research methods and principles
Over the past decade, many research and development works have been produced to create new approaches and methods for recognizing emotions. Choosing an approach depends on how researchers analyze and classify emotions. Of the many approaches, two main ones can be distinguished: discrete and constructivist [5].
The discrete approach assumes the presence of stable patterns of emotions in humans as an object of study. According to this approach, there is a certain neural network structure of the brain and a corresponding pattern in biological signals for each emotion.
The constructivist approach assumes that emotions are constructed in the object of study depending on behavioral or value models. This approach involves several measurements of EEG or ECG parameters in order to create an effective basis for studying and classifying emotions in each individual subject. The most indicative model of this approach is the descriptive model (VAD - "Valence-Arousal-Dominance") [6].
Currently, EEG is a reliable and easy to implement technology used to measure brain activity, and ECG is a technology that shows the effect of brain activity on the functional response of the rest of the body. Emotion detection using EEG and ECG signals is carried out in several stages, performed sequentially. Traditionally, these steps include removing interference from signals, extracting temporal or spectral features from oscillograms or signal spectrograms, and then developing a multi-class classification strategy.
Traditional methods for recognizing emotional responses involve extracting emotion traits from a fixed group of the same EEG and ECG channels for all study subjects. However, in some studies, it has been recorded that with the same emotional response, different brain regions can be activated in different study objects, that is, the emotional response can change from person to person and from one emotional state to another [3].
The algorithm developed by the authors of this article for recognizing the psychophysiological state of a person in real time includes the following steps:
1. Measurement of emotional and physiological responses by EEG and ECG in several subjects under different conditions;
2. Use of fast Fourier transform methods [7], [8] and wavelet analysis [9] to extract spectral information from EEG and ECG signals;
3. Rationale and choice of methods for analyzing EEG and ECG signals demonstrating significant changes in brain activity during emotional states;
4. Highlighting standard patterns for each emotional response.
Main results
The implementation of the first stage of the algorithm depends on the used technical means for fixing signals, as well as on the method of identifying the emotional state of the individual. This stage is currently under development and testing.
Pilot experiments were carried out with the participation of a group of conditionally healthy students in the amount of 55 people using a device for fixing photoplethysmograms MAX30102 I2C [10], which showed that the signals of the photoplethysmograph cannot help solve the tasks set in the work (since such experiments can only find out the pulse rate of the subjects). Funds are being sought for the purchase of equipment for taking digital readings of the ECG and EEG, the analysis
of which, as will be shown below, allows you to diagnose the emotional state of the individual. It is planned to make a variant search for the best option for taking readings (single or combined use of ECG and EEG devices).
To demonstrate the 2 and 3 stages of the software implementation of the ECG signal processing and decryption algorithm using the fast Fourier transform method with the created Python program, the heart beat sound from the Internet in mp3 format was taken, which was then converted into a wav file. Figure 1 shows the oscillogram of the ECG signal read from the wav format, which shows the dependence of its amplitude in conditional dimensionless units of time in seconds.
-g 30000 -3
zoo oo -
a.
< 10000 -o --10000 --20000 --30000 -
0 1 2 3 4 5 6
Time, sec.
Figure 1 - Electrocardiogram read from wav format DOI: https://doi.org/10.23670/IRJ.2023.133.40.!
Figure 2 shows the output of the ECG signal spectrogram using the commands of the function library Matplotlib.pyplot obtained by the fast Fourier transform method. The amplitude of the spectral components of the signal is expressed in conventional dimensionless units.
Figure 2 - The result of obtaining an electrocardiogram spectrum DOI: https://doi.org/10.23670/IRJ.2023.133.40.2
The obtained ECG signal spectrogram shows a significant increase in amplitude at frequencies of 22 Hz, 44 Hz, 66 Hz, 88 Hz and 110 Hz. However, the amplitude of frequency components with a frequency greater than 110 Hz in the ECG spectrum is small. Thus, as a result of analysis by the fast Fourier transform method, it was possible to identify 5 basic multiples of each other harmonics.
Discussion
When processing ECG or EEG signals using the selected method in real time, the frequencies and amplitudes of harmonics will differ depending on the psychophysiological states of the individual. By calculating the number of harmonics, their frequencies and amplitude ratios, it is planned to find a pattern of significant changes in the biological activity of the nervous system during various emotional states, as well as, perhaps, to identify deviations from the normal state for use in the computer program under development, which should be used as an assistant for diagnosing various diseases.
We consider the use of artificial neural networks (ANN) as an adaptive method for solving this problem to be scientifically justified due to the obvious change in the frequency and amplitude characteristics of the individual's cardiogram obtained in real time [11]. We will use the harmonics and frequencies obtained during spectral analysis as the initial network data. In this case, to use ANN, work out the following steps:
1. Prepare a sample of data for training an artificial neural network and receive synchronized ECG signals [12];
2. Create a software package for displaying signals and calculating parameters that will subsequently be used to create a neural network;
3. Create a neural network to obtain emotion values according to the VAD model - "Valence-Arousal-Dominance";
4. Analyze the results obtained by the artificial neural network;
5. Recognize the psycho-emotional state of a person by the results obtained.
In order to identify various anomalies of the psycho-emotional state of a person, as well as collect statistical data, it is planned to continuously monitor the state of an individual using wearable gadgets.
Conclusion
The results of the study can be used for medical purposes as an assistant to make a correct diagnosis to the patient. At the same time, the algorithms tested by the experiment will serve as the basis for creating a mobile system for automatic assessment and monitoring of the psycho-emotional state of an individual. This system will consist of a body sensor for recording the state and a program with an algorithm for processing continuous measurement results for a smartphone. As a test of the use of software for the first stage of research, we have developed a layout of an application for a smartphone through which the user could view the measurement results of biological signals and, responding to them, adjust his emotional state. In this case, a photoplethysmograph built into a smart watch is used as a heart rate sensor. It is also planned to use a wearable ECG analyzer "stitched" by the software created by the authors.
Конфликт интересов
Не указан.
Рецензия
Все статьи проходят рецензирование. Но рецензент или автор статьи предпочли не публиковать рецензию к этой статье в открытом доступе. Рецензия может быть предоставлена компетентным органам по запросу.
Conflict of Interest
None declared.
Review
All articles are peer-reviewed. But the reviewer or the author of the article chose not to publish a review of this article in the public domain. The review can be provided to the competent authorities upon request.
Список литературы / References
1. Emotion: Theory, Research, and Experience. Vol. 1. Theories of Emotion / Ed. by R. Plutchik, H. Kellerman. — London: Academic Press, 1980. — 399 p.
2. Дмитриева А.Д. Влияние эмоций на здоровье и деятельность человека / А.Д. Дмитриева, М.В. Подплетько // Психология развития и образования. — 2020. — № 1. — С. 16-18.
3. Баррет Л.Ф. Как рождаются эмоции. Революция в понимании мозга и управлении эмоциями / Л.Ф. Баррет. — М.: Манн, Иванов и Фербер, 2018. — 649 с.
4. Hasnul M.A. Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare — A Review / M.A. Hasnul, S. Alelyani, M. Mohana et al. // Sensors. — 2021. — DOI: 10.3390/s21155015.
5. Карташева А.А. Подходы к распознаванию эмоций в интеллектуальных системах / А.А. Карташева // Технологос. — 2020. — № 2. — С. 15-24.
6. Bälan O. Emotion Classification Based on Biophysical Signals and Machine Learning Techniques / O. Bälan, G. Moise, L. Petrescu et al. // Symmetry. — 2020. — DOI: 10.3390/sym12010021.
7. Ильичев В.Ю. Гармонический анализ сложного сигнала колебаний газотурбинного электроагрегата / В.Ю. Ильичев // Заметки Ученого. — 2021. — № 12. — C. 82-86.
8. Ильичев В.Ю. Применение методов компьютерного статистического анализа для прогнозирования потребления электрической энергии / В.Ю. Ильичев, И.В. Чухраев, Е.А. Юрик // Информационно-измерительные и управляющие системы. — 2020. — Т. 18. — № 2. — С. 24-32.
9. Ильичев В.Ю. Вейвлет-анализ непериодических сигналов с использованием специальных библиотек Python / В.Ю. Ильичев, Ю.М. Жукова, И.В. Шамов // Заметки ученого. — 2021. — № 13. — С. 43-47.
10. Измеритель пульса и уровня кислорода в крови на MAX30102 I2C // Амперкот. — URL: https://amperkot.ru/msk/catalog/izmeritel_pulsa_i_urovnya_kisloroda_v_krovi_na_max30102_i2c-38971151.html (дата обращения: 22.04.23).
11. Ильичев В.Ю. Обработка данных с использованием глубокого обучения генеративно-состязательной нейронной сети (gan) / В.Ю. Ильичев, И.В. Чухраев // Радиотехника. — 2021. — T. 23. — № 5. — С. 51-56.
12. Ильичев В.Ю. Обработка статистических данных методом глубокого обучения с использованием модуля keras / В.Ю. Ильичев, Е.А. Юрик // Научное обозрение. Технические науки. — 2020. — № 5. — C. 16-20.
Список литературы на английском языке / References in English
1. Emotion: Theory, Research, and Experience. Vol. 1. Theories of Emotion / Ed. by R. Plutchik, H. Kellerman. — London: Academic Press, 1980. — 399 p.
2. Dmitrieva A.D. Vlijanie jemocij na zdorov'e i dejatel'nost' cheloveka [The Influence of Emotions on Human Health and Performance] / A.D. Dmitrieva, M.V. Podplet'ko // Psihologija razvitija i obrazovanija [Developmental and Educational Psychology]. — 2020. — № 1. — P. 16-18. [in Russian]
3. Barret L.F. Kak rozhdajutsja jemocii. Revoljucija v ponimanii mozga i upravlenii jemocijami [How Emotions are born. A Revolution in Understanding the Brain and Managing Emotions] / L.F. Barret. — M.: Mann, Ivanov and Ferber, 2018. — 649 p. [in Russian]
4. Hasnul M.A. Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare — A Review / M.A. Hasnul, S. Alelyani, M. Mohana et al. // Sensors. — 2021. — DOI: 10.3390/s21155015.
5. Kartasheva A.A. Podhody k raspoznavaniju jemocii v intellektual'nyh sistemah [Approaches to Emotion Recognition in Intelligent Systems] / A.A. Kartasheva // Technologos. — 2020. — № 2. — P. 15-24. [in Russian]
6. Bälan O. Emotion Classification Based on Biophysical Signals and Machine Learning Techniques / O. Bälan, G. Moise, L. Petrescu et al. // Symmetry. — 2020. — DOI: 10.3390/sym12010021.
7. Il'ichev V.Ju. Garmonicheskij analiz slozhnogo signala kolebanij gazoturbinnogo jelektroagregata [A Harmonic Analysis of a Complex Oscillation Signal of a Gas Turbine Electric Unit] / V.Ju. Il'ichev // Zametki Uchenogo [Notes of the Scientist]. — 2021. — № 12. — P. 82-86. [in Russian]
8. Il'ichev V.Ju. Primenenie metodov komp'juternogo statisticheskogo analiza dlja prognozirovanija potreblenija jelektricheskoj jenergii [Application of Computer Based Statistical Analysis to Forecast Electricity Consumption] / V.Ju. Il'ichev, I.V. Chuhraev, E.A. Jurik // Informacionno-izmeritel'nye i upravljajushhie sistemy [Information Measurement and Control Systems]. — 2020. — Vol. 18. — № 2. — P. 24-32. [in Russian]
9. Il'ichev V.Ju. Vejvlet-analiz neperiodicheskih signalov s ispol'zovaniem special'nyh bibliotek Python [A Wavelet Analysis of Non-periodic Signals Using Special Python Libraries] / V.Ju. Il'ichev, Ju.M. Zhukova, I.V. Shamov // Zametki uchenogo [Notes of the Scientist]. — 2021. — № 13. — P. 43-47. [in Russian]
10. Izmeritel' pul'sa i urovnja kisloroda v krovi na MAX30102 I2C [Pulse and blood oxygen level meter on MAX30102 I2C] // Amperkot. — URL: https://amperkot.ru/msk/catalog/izmeritel_pulsa_i_urovnya_kisloroda_v_krovi_na_max30102_i2c-38971151.html (accessed: 22.04.23). [in Russian]
11. Il'ichev V.Ju. Obrabotka dannyh s ispol'zovaniem glubokogo obuchenija generativno-sostjazatel'noj nejronnoj seti (gan) [Data Processing Using Deep Learning Generative Adversarial Neural Network (gan)] / V.Ju. Il'ichev, I.V. Chuhraev // Radiotehnika [Radio Engineering]. — 2021. — Vol. 23. — № 5. — P. 51-56. [in Russian]
12. Il'ichev V.Ju. Obrabotka statisticheskih dannyh metodom glubokogo obuchenija s ispol'zovaniem modulja keras [Statistical Data Processing by Deep Learning Using the keras Module] / V.Ju. Il'ichev, E.A. Jurik // Nauchnoe obozrenie. Tehnicheskie nauki [Scientific Review. Technical Sciences]. — 2020. — № 5. — P. 16-20. [in Russian]
