Multsurv: a multimodal deep learning model for hospitalized patients survival analysis in the contexto of a pandemic

Abstract

BACKGROUND: Respiratory infectious diseases represent a major challenge in modern society. We recently faced the most significant public health challenge of the last century. Severe Acute Respiratory Syndrome Coronavirus 2 has overwhelmed almost all health systems worldwide, highlighting pre-existing weaknesses. The heterogeneity of COVID-19 clinical manifestations has made it challenging to manage hospitalized patients, making it crucial to identify those at greatest risk, especially for eciently allocating vital resources. Unlike past pandemics, hospitalized patients are currently monitored continuously and through di↵erent modalities. These data generate large longitudinal and multimodal datasets in health institutions. In this context, data-driven solutions can support clinical decisions and provide new tools for risk management of hospitalized patients during pandemics. OBJECTIVE: Therefore, we propose integrating clinical, laboratory, and chest X-ray imaging features into a survival analysis model for hospitalized patients with COVID-19. With the model, we aim to combine multimodal and longitudinal data to capture the dynamic nature of COVID-19 and provide an explainable hazard function. METHODOLOGY: The methodology involves the proposition and development of the model. The model is divided into five main components: (i) pre-processing; (ii) feature encoders; (iii) temporal attention; (iv) CheXReport; and (v) multitask networks. The pre-processing component is responsible for data cleaning, outlier removal, variable selection, and image processing. In the feature encoders, the categorical and continuous data are transformed into a vector of embeddings that capture the complex and non-linear relationships between the variables. Then, based on the embeddings up to the current time instant, we extract a temporal context vector using temporal attention. The CheXReport component processes the patient’s X-ray images using a fully-transformers architecture, which integrates visual features with the textual elements of the reports. Finally, all feature vectors are concatenated to be processed in the multitask networks, a set of neural networks that allow the model to capture the specific characteristics of each risk. RESULTS: To evaluate the model performance, we used an incremental ablation study. We use the public datasets PBC2, MIMIC-CXR, Curated Dataset for COVID-19, and a private dataset. Then, we compare the results of the MultSurv model with the state of the art. The results obtained demonstrate that the MultSurv outperforms all reference architectures, with a C-index of 0.723 ± 0.008 for t = 1 and t = 1, and 0.695 ± 0.003 for t = 7 and t = 7. CONCLUSION: The main scientific contribution of this study is the proposal of a multimodal model for processing dynamic and longitudinal data in survival analysis in the context of COVID-19. Furthermore, the MultSurv model offers a tool to support patient prioritization in pandemic scenarios. Finally, the application of the model can be adapted to di↵erent clinical contexts, extending beyond COVID-19.