Summary
Background
Non-invasive respiratory strategies (NIRS) including high-flow nasal cannula (HFNC) and non-invasive ventilation (NIV) have become widely used in patients with COVID-19 who develop acute respiratory failure. However, use of these therapies, if ineffective, might delay initiation of invasive mechanical ventilation (IMV) in some patients. We aimed to determine early predictors of NIRS failure and develop a simple nomogram and online calculator that can identify patients at risk of NIRS failure.
Methods
We did a retrospective, multicentre observational study in 23 hospitals designated for patients with COVID-19 in China. Adult patients (≥18 years) with severe acute respiratory syndrome coronavirus 2 infection and acute respiratory failure receiving NIRS were enrolled. A training cohort of 652 patients (21 hospitals) was used to identify early predictors of NIRS failure, defined as subsequent need for IMV or death within 28 days after intensive care unit admission. A nomogram was developed by multivariable logistic regression and concordance statistics (C-statistics) computed. C-statistics were validated internally by cross-validation in the training cohort, and externally in a validation cohort of 107 patients (two hospitals).
Findings
Patients were enrolled between Jan 1 and Feb 29, 2020. NIV failed in 211 (74%) of 286 patients and HFNC in 204 (56%) of 366 patients in the training cohort. NIV failed in 48 (81%) of 59 patients and HFNC in 26 (54%) of 48 patients in the external validation cohort. Age, number of comorbidities, respiratory rate–oxygenation index (ratio of pulse oximetry oxygen saturation/fraction of inspired oxygen to respiratory rate), Glasgow coma scale score, and use of vasopressors on the first day of NIRS in the training cohort were independent risk factors for NIRS failure. Based on the training dataset, the nomogram had a C-statistic of 0·80 (95% CI 0·74–0·85) for predicting NIV failure, and a C-statistic of 0·85 (0·82–0·89) for predicting HFNC failure. C-statistic values were stable in both internal validation (NIV group mean 0·79 [SD 0·10], HFNC group mean 0·85 [0·07]) and external validation (NIV group value 0·88 [95% CI 0·72–0·96], HFNC group value 0·86 [0·72–0·93]).
Interpretation
We have developed a nomogram and online calculator that can be used to identify patients with COVID-19 who are at risk of NIRS failure. These patients might benefit from early triage and more intensive monitoring.
Funding
Ministry of Science and Technology of the People’s Republic of China, Key Research and Development Plan of Jiangsu Province, Chinese Academy of Medical Sciences.
Introduction
Non-invasive respiratory strategies (NIRS), which include high-flow nasal cannula (HFNC) and non-invasive ventilation (NIV), are now widely used in these patients.
,
,
NIRS can decrease the need for invasive mechanical ventilation (IMV) in patients with acute respiratory failure,
,
but patients who do not respond to NIRS have poor outcomes.
,
,
This lack of response might be particularly important in patients with COVID-19, since the availability of health-care services can be strained during pandemics. Therefore, early identification of patients with COVID-19 who are unlikely to respond to treatment with NIRS would be beneficial.
,
,
,
,
Similarly, a number of clinical and oxygenation variables, including no clinical improvement in oxygenation or decrease in respiratory rate, have been associated with HFNC failure and subsequent need for IMV;
,
however, most of these variables were of limited value in identifying patients who would require subsequent intubation. In patients with acute respiratory failure and pneumonia, the respiratory rate–oxygenation (ROX) index, based on oxygen saturation measured by pulse oximetry (SpO2), fraction of inspired oxygen (FiO2), and respiratory rate, can help to identify the risk of NIRS failure and intubation.
,
However, which indicators are useful in identifying patients with COVID-19 with a high risk of NIRS failure is unknown. The objective of this study was to develop and validate a simple nomogram and online calculator for predicting the risk of NIRS failure in patients with COVID-19 presenting with acute respiratory failure.
Evidence before this study
We searched MEDLINE and medRxiv for papers published up to Nov 8, 2020, without language restrictions, using the search terms: ((COVID-19 or SARS-CoV-2 or novel coronavirus) and (acute hypoxemic respiratory failure or ARDS or ALI) and (non-invasive ventilation [NIV] or high-flow nasal cannula oxygen [HFNC]) and (nomogram or predictor or prediction)). Many studies have shown that various forms of non-invasive respiratory support (NIRS), including NIV or HFNC, are widely used in patients with COVID-19 with acute respiratory failure. In several studies of other respiratory conditions, NIRS decreased the need for invasive mechanical ventilation (IMV) in some patients with hypoxaemic respiratory failure, but patients who did not respond to these therapies had poor outcomes. As such, an important aim should be early identification of patients at high risk of NIRS failure (and thus in need of intubation or IMV) to prevent death. In hypoxaemic patients without COVID-19, an easily implemented scale that includes heart rate, acidosis, consciousness, oxygenation, and respiratory rate appeared to be effective in predicting NIV failure. The respiratory rate–oxygenation index (defined as the ratio of pulse oximetry oxygen saturation/fraction of inspired oxygen to respiratory rate) can help to identify the risk of HFNC failure and subsequent intubation in patients with acute respiratory failure and pneumonia. However, no indicators have been developed to identify the risk of NIRS failure in patients with COVID-19.
Added value of this study
Our results are consistent with and build on previous prediction models in non-COVID-19 patients. To our knowledge, this study is the first to develop a simple nomogram and online calculator that can identify patients with COVID-19 and acute respiratory failure with a high probability of NIRS failure.
Implications of all the available evidence
By identifying patients at risk of NIRS failure, clinicians could ascertain those who might benefit from early monitoring and early interventions. However, any benefits from such a strategy, which might include early intubation, require confirmation in clinical trials.
Methods
Study design and populations
and acute respiratory failure (defined as partial pressure of arterial oxygen (PaO2)/FiO2 ≤300 mm Hg) treated with HFNC or NIV and admitted to intensive care units (ICUs) as a consequence of acute respiratory failure. Patients with missing data for at least one NIRS failure predictor in the nomogram after univariable analysis to identify relevant predictors were excluded. Model development and internal validation involved a cohort of patients from 21 hospitals. A second cohort from two different hospitals was used for external validation of the model.
The two hospitals selected for external validation were general medical hospitals accepting patients with COVID-19. One of the hospitals (Wuhan Third Hospital, Wuhan, China) had 650 beds, 35 ICU beds, and 1684 hospitalised patients with COVID-19 during the study period. The second hospital (Wuhan Red Cross Hospital, Wuhan, China) had 400 beds, 15 ICU beds, and 1182 hospitalised patients with COVID-19. The 21 hospitals used for development of the model had a median of 1400 beds (IQR 493–2609) and 23 ICU beds (13–43). Overall, each of the 21 hospitals had a median of 554 patients (IQR 229–1021) admitted with COVID-19. Data were collected throughout the enrolment period for both cohorts.
Data collection and study outcomes
Vital signs, Glasgow coma scale score,
sequential organ failure assessment (SOFA) score,
clinical respiratory variables including respiratory rate, FiO2, SpO2, and blood gases were recorded on the first day of HFNC or NIV use (after HFNC or NIV was started). ROX index was defined as the ratio [(SpO2/FiO2)/respiratory rate], in units of breaths per min. We recorded the dates of initiation, any switching of ventilation therapy, and the duration of HFNC, NIV, and IMV. Use of vasopressors, steroids, and antivirals, and 28-day mortality after ICU admission were also recorded, as well as negative nucleic acid testing for SARS-CoV-2 within 28 days after ICU admission. NIV was delivered by face mask or nasal mask under bi-level positive pressure ventilation. Patients who received HFNC followed by NIV were included in the HFNC group, and patients who received NIV followed by HFNC were included in the NIV group.
The primary outcome was NIRS failure, defined as the subsequent use of IMV or death within 28 days after ICU admission. Death within 28 days after ICU admission was a secondary outcome. Patients who transitioned from HFNC to NIV or from NIV to HFNC, but did not receive IMV or die, were considered as being successfully treated with NIRS.
Statistical analysis
S0(t) is the baseline survival function, estimated from the data. Regression coefficients were used to construct the variable axes in the nomogram and S0 was used in the translation from total points to predicted probability.
The Akaike information criterion was calculated to assess the goodness of fit of the model. Cox-Snell R2 and Nagelkerke R2 were calculated to assess the prediction accuracy of the logistic regression model.
Cross-validation was applied to internally validate the stability of the model, by randomly splitting the patients in the training cohort into ten equal samples. Nine of these samples were used to construct logistic regression models and the model coefficients were applied to the remaining sample. This process was repeated 10 times and the mean C-statistic plus SD values corresponding to each iteration were calculated.
To assess external validity, the model was applied to our independent dataset from two hospitals. External validity of the model was assessed with the C-statistic, calibration, and decision curve analysis in patients treated with HFNC and NIV separately.
All statistical analyses were done with RStudio (version 1.2.5019), and a p value of less than 0·05 was considered to indicate statistical significance.
Role of the funding source
The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.
Results
Table 1Demographic, respiratory, and treatment variables in patients with NIRS failure or success
Characteristics are summarised as median (IQR) or frequency (%). NIRS=non-invasive respiratory support. PaO2=partial pressure of arterial oxygen. FiO2=fraction of inspired oxygen. PaCO2=partial pressure of arterial carbon dioxide. NIV=non-invasive ventilation. HFNC=high-flow nasal cannula. IMV=invasive mechanical ventilation. ICU=intensive care unit.
Table 2Factors associated with non-invasive respiratory support failure in univariable and multivariable analyses
Figure 1Characteristics in the nomogram to predict probability of NIRS failure in patients with severe acute respiratory syndrome coronavirus 2 pneumonia
Patient prognostic values are located on the axis of each variable; a line is then drawn upwards at a 90° angle to determine the number of points for that particular variable. The sum of these numbers is located on the total score axis, and a line is drawn at a 90° angle downward to the NIRS failure risk axis to determine the likelihood of failure of non-invasive respiratory therapies. Alternatively, failure risk can be ascertained from the online calculator. Vasopressor use was represented on the axis at an arbitrary value of 1 (no use=0). NIRS=non-invasive respiratory support.
Probability of NIRS failure was calculated as:
Table 3C-statistics for the nomogram and model variables in the training and external validation cohorts
C-statistic=concordance statistic. NIRS=non-invasive respiratory support. NIV=non-invasive ventilation. HFNC=high-flow nasal cannula.
Figure 2Calibration curves for the nomogram
The calibration method with bootstrapping was used to illustrate the association between actual NIRS failure and predicted NIRS failure. Calibration plots show the apparent (actual), bias-corrected (adjusted), and ideal (100% agreement) curves with bootstrapping samples. Bootstrapping involved 1000 repetitions. Nomogram-predicted probability of NIRS failure is plotted on the x-axis; the observed probability of NIRS failure is plotted on the y-axis. HFNC=high-flow nasal cannula. NIV=non-invasive ventilation. NIRS=non-invasive respiratory support.
Discussion
In this study, we developed and validated a nomogram and online calculator for the early prediction of NIRS failure in patients with COVID-19. The nomogram, based on age, number of comorbidities, ROX index, Glasgow coma scale score, and use of vasopressors on day 1 of NIRS, had a discriminatory ability (C-statistic) of 0·84 (95% CI 0·81–0·87) in predicting NIRS failure. Patients in whom NIRS fails have a high risk of death. Thus, early prediction of NIRS failure could help clinicians to appropriately allocate critical care resources, and identify high-risk patients for entry into clinical trials.
,
or HFNC
(or NIRS generally
) is associated with worse outcomes, such as increased mortality, in patients with acute respiratory failure. Predicting the outcome of NIRS is particularly important in patients with COVID-19, given the limited resources available during the pandemic. Although previous studies have suggested that the criteria for initiating HFNC or NIV and their outcomes might differ,
we found that most independent risk factors for NIRS failure overlapped in the HFNC and NIV groups in the training cohort (appendix pp 12–13). Indeed, 26% of patients in the training cohort received both HFNC and NIV at various times in the course of their disease. Furthermore, the discriminatory ability of the HFNC and NIV models was similar to the final models in each cohort. Therefore, we analysed HFNC and NIV together as NIRS to make the nomogram easier to use. We verified a previous predictive scale for NIV failure in non-COVID-19 patients (based on heart rate, acidosis, consciousness, oxygenation, and respiratory rate)
with our data in the training cohort and found a C-statistic of 0·83 (95% CI 0·81–0·87), which was similar to our final model. However, PaO2/FiO2 was available only in a portion of our patients. The need for these blood gases and manual score calculation might restrict application of the previously proposed scale during the COVID-19 pandemic.
,
and NIV (39–50%).
,
,
,
The high failure rate in the current study could be due in part to the severity of hypoxaemia (median PaO2/FiO2 116 mm Hg [IQR 67–215) on day 1 of NIRS in the training cohort) as compared with that on day 1 of previous studies (with HFNC, 149 mm Hg [SD 72]
to 160 mm Hg [64];
and with NIV, 157 mm Hg [89]
). Blood gases were only available in 64% of patients receiving NIRS in the training cohort and 69% of patients in the validation cohort. Thus, low PaO2/FiO2 values in the present study might be partially due to the fact that only patients with severe COVID-19 were likely to have these measurements taken due to limited resources. This could have led to underestimation of average PaO2/FiO2. Another explanation could be insufficient supply of invasive ventilators during the pandemic, leading to the use of non-invasive approaches in patients with low PaO2/FiO2 who might otherwise have been intubated and ventilated.
,
,
However, patients in whom NIRS failed had substantially higher mortality than that reported previously in other respiratory conditions after NIV or HFNC failure.
,
,
In both the training and validation cohorts, mortality rate was significantly higher in the NIV group than in the HFNC group. Given the criteria for initiating HFNC and NIV, it is not unexpected that patients in the NIV group had more severe acute respiratory failure and higher mortality than those in the HFNC group.
,
,
In the present study, our nomogram included ROX as a continuous variable to predict the risk of NIRS failure in patients with COVID-19. Age has been shown to be an independent risk factor for death in patients with COVID-19 in a previous ICU cohort (n=344),
but it has not been commonly reported in association with NIRS failure. In our study, age was an independent risk factor for NIRS failure. The predictive accuracy of age alone in our study was represented by a C-statistic of 0·68 (95% CI 0·64–0·72) in the training cohort and 0·71 (0·60–0·83) in the validation cohort, showing that age itself had moderate discriminative power. In agreement with previous research on predicting NIV failure in hypoxaemic patients,
we also found that Glasgow coma scale was an independent risk factor for NIRS failure, but with low discriminative power.
we found that the number of comorbidities was independently associated with NIRS failure. Although only 7% of patients in our training cohort and 5% in our validation cohort received vasoactive drugs on the first day of NIRS, our results are consistent with previous findings, that use of vasoactive drugs was associated with NIRS failure.
Median duration from symptom onset to hospital admission in our cohort of ICU patients was 9 days, which was higher than previously reported in 1590 hospitalised patients (mean 4·4 to 4·7 days).
However, time between symptom onset and admission was not independently associated with NIRS failure in multivariable analysis. With respect to clinical utility, decision curve analysis also indicated that the nomogram was feasible in clinical practice, reflected by the positive net benefit associated with application of the nomogram over a broad range of threshold probabilities of NIRS failure (0·00 to 0·94 in the training cohort, and 0·00 to 0·89 in the validation cohort).
Second, we only enrolled patients admitted in January and February, 2020, when medical resources were overwhelmed by the surge of COVID-19 cases. The median duration from symptom onset to hospital admission or ICU admission and the practice of respiratory support might be different after a pandemic period, or in other countries. Third, agitation and intolerance to masks might have had prognostic implications against the tested parameters (eg, ROX index) in determining failure of NIRS. However, these data were unavailable in our retrospective study. Finally, although our nomogram and online calculator can identify patients at risk of NIRS failure, our study cannot determine whether an alternative management strategy for these patients would improve outcomes.
In conclusion, our nomogram and online calculator are simple to use and able to predict the risk of failure in patients with COVID-19 treated with HFNC and NIV. The nomogram and online calculator can be used to identify patients with a high probability of NIRS failure. These patients might benefit from early triage and more intensive monitoring. The benefits of such a strategy, which might include early intubation, would require confirmation in randomised control trials.
LL, HQ, YYa, HC, and BD contributed to study conception and design. WW, JX, HC, SL, JL, YYu, ZT, RZ, HH, YYa, LL, MH, and XL contributed to data acquisition. JX and HC were independent of the funders and had access to and verified the underlying data. LL, HC, EF, ASS, JX, and YYu contributed to data analysis and interpretation. LL, HC, EF, ASS, HQ, and BD contributed to drafting of the manuscript and revision for important intellectual content. All authors had full access to all the data in the study. All authors revised the manuscript and approved the final version before submission. The corresponding author had final responsibility for the decision to submit for publication.
EF reports personal fees from ALung Technologies, Fresenius Medical Care, and MC3 Cardiopulmonary, outside the submitted work. ASS reports personal fees from Baxter and consultancy fees from Xenios, outside the submitted work. All other authors declare no competing interests.
