Introduction

A significant portion of patients admitted to Intensive Care Units (ICUs) undergo different type of surgical procedures [1,2].These procedures span a broad spectrum of specialties, from routine outpatient treatments to complex interventions requiring hospitalization. Effective postoperative pain management is crucial to expedite recovery and enhance patient outcomes. Historically, opioids have been instrumental in providing potent analgesic effects to alleviate postoperative pain and improve patient comfort [3,4]. However, the widespread use of opioids for postoperative pain relief has contributed to the escalating opioid crisis, posing significant public health challenges. Inappropriate or excessive opioid use can lead to adverse outcomes such as dependency, addiction, overdose, and fatalities [5-11].Additionally, opioid-related side effects like respiratory depression, nausea, constipation, and sedation can compromise patient safety and impede recovery if not adequately managed [12-15].  Therefore, while opioids remain a vital component of postoperative pain management, it is imperative to balance their benefits and risks, implementing comprehensive strategies to mitigate opioid-related harm and optimize patient care.  Accurate prediction of postoperative opioid use is essential for achieving a delicate balance between optimizing pain control and mitigating associated risks. Precise forecasting of opioid consumption following surgery allows healthcare providers to tailor pain management strategies to individual patient needs, reducing the risk of overprescription while ensuring adequate pain relief [16-19].  

However, accurate prediction presents several challenges. Variations in patient responses to opioids, differences in pain perception, age, sex, and the complexity of surgical procedures can all influence opioid requirements, complicating individualized prediction [11, 20-24]. Additionally, the multifaceted nature of postoperative pain, involving both nociceptive and neuropathic components, further complicates opioid consumption prediction [25-27]. Therefore, the potential for opioid-related adverse effects and the risk of misuse and addiction underscores the importance of precise individualized prediction to optimize pain management outcomes while minimizing harm [16-18]. Overcoming these challenges necessitates advanced predictive modeling techniques [28-31].By integrating predictive analytics with clinical expertise, healthcare providers can enhance their ability to accurately forecast postoperative opioid use, ultimately improving patient safety and optimizing postoperative pain control.

Existing predictive models for postoperative opioid consumption often adopt parametric techniques, which necessitate restrictive assumptions by explicitly specifying the functional association between predictors and postoperative opioid consumption [22,28,32]. These restrictive assumptions may not hold in clinical practice and are unverifiable [33,34]. Misspecification of the functional association format can lead to inaccurate predictions. Conversely, many machine learning methods developed in recent years relax many of these restrictive assumptions [35].In this study, we investigated the effectiveness of four commonly used machine learning models in predicting postoperative opioid consumption. Specifically, we compared Support Vector Machine (SVM), Extreme Gradient Boosting (XGBoost), Random Forest (RF), and Deep Neural Network (DNN) based on patients’ demographics, clinical, preoperative, and operative features [36-42]. While these machine learning models are robust in exploring complex associations, they lack transparency in evaluating each individual feature’s impact on postoperative opioid consumption due to the abstract algorithms used [43-45].  Identifying important features associated with postoperative opioid consumption helps strengthen our understanding of postoperative opioid consumption mechanisms and improves postoperative pain management for ICU surgery patients. To address transparency limitation of machine learning models, we adopted the permutation-based feature importance test (PermFIT) procedure [46].  The overall goal of this paper is to develop and validate a machine learning framework for accurate personalized postoperative opioid need prediction and identify the associated important features under complex associations based on ICU surgical patients’ Electronic Health Records (EHRs) captured in clinical practice.

Materials and Methods

Study Design

This prognostic study followed TRIPOD guidelines with data included in this study extracted from the Medical Information Mart for Intensive Care (MIMIC-III) database [47].  This large database includes the EHRs of patients admitted to critical care units at the Beth Israel Deaconess Medical Center between 2001 and 2012.  We developed and validated a robust predictive model under complex association relationship for accurate postoperative opioid consumption prediction in ICU surgery patients by investigating a set of commonly used machine learning models. Additionally, significant factors associated with postoperative opioid consumptions for each machine learning model were identified under the PermFIT framework [46], providing an in-depth understanding of postoperative opioid consumption mechanisms for ICU patients.  More importantly, evaluation of each individual feature’s impact on postoperative opioid consumption will enable each of the black-box machine learning model to be interpretable, i.e., our feature importance identification for machine learning models was derived from rigorous statistical inference which is not available in existing postoperative opioid prediction models.

Study Population

The MIMIC-III database contains EHRs of over 40,000 patients admitted to the ICU, with 17,611 undergoing surgical procedures[47]. Among these surgical patients, 7,192 had documented opioid consumptions during their hospital stay.  To ensure the reliability of our dataset, rigorous preprocessing procedures were implemented, revealing two distinct modalities for intravenous opioid administration: continuous infusion and discreet bolus dose. While some patients exclusively received opioids via continuous infusion (n=619), a large proportion patients underwent intravenous push administration (n=7,106). Additionally, a subset received opioids through both methods (n=533). Given pharmacokinetic differences, our analysis focused solely on opioids administered via intravenous push. Although oral opioids like Oxycodone are common in clinical practice, their absence in our dataset led to the exclusion of oral opioid data, resulting in a cohort of 7,106 patients receiving intravenous opioids. Refinement procedures eliminated aberrant observations, including extreme opioid intake and missing values. Furthermore, to avoid redundancy, in cases where patients had multiple hospital admissions (HADM_ID), a random selection process was employed to retain only one HADM_ID per patient. Our analysis ultimately included 4,912 surgery patients with complete demographic, clinical, preoperative, operative, and postoperative features.  Detailed data processing procedures are presented in Figure 1.

Figure 1: Data Extraction and Modeling Architecture.

Opioid Consumption Derivation and Covariates

The average daily postoperative opioid consumption was derived and calculated as the intake amount of four opioid drugs frequently used in early recovery period after surgery: morphine, hydromorphone, fentanyl, and meperidine via intravenous push route, converted to morphine equivalent dose [48,49] , using Micrograms (mcg) as the unit of measurement. 25 Variables, Including Demographic (e.g., age, sex, ethnicity), preoperative (e.g., prior medical history), operative (e.g., surgery types), and other clinical features, were incorporated as input features of each machine learning model for predicting the average daily postoperative opioid consumption.

Statistical Analysis

Build Interpretable Machine Learning Models for Opioid Use Prediction

A 10-fold cross-validation strategy was adopted to evaluate the prediction performance of each machine learning model in predicting the postoperative opioid consumption as depicted in Figure 1. Specifically, by randomly partitioning the dataset into ten distinctive subsets, each fold of the 10 subsets was alternatively used as a testing set, while the rest of 9 folds were used to train the machine learning model. Four commonly used machine learning models including SVM, RF and XGBoost, and DNN were developed and validated based on MIMIC-III EHR data. To identify the significant features associated with postoperative opioid consumption for each machine learning model, we adopted the PermFIT framework [37]. To comprehensively evaluate the model performance in predicting the postoperative opioid consumption, we employed two evaluation metrics: Mean Square Error (MSE) - a quantitative measure of the average squared difference between predicted daily postoperative opioid consumptions and actual daily postoperative opioid consumptions, and Pearson Correlation Coefficient (PCC) - measures the strength and direction of the linear relationship between predicted and actual daily postoperative opioid consumptions.

Tuning Machine Learning Models

Machine learning models were meticulously tuned. The SVM model's tuning involved optimizing gamma (from 10^-4 to 1) and cost parameters (1 to 100) through a cross-validation strategy. For the RF model, 1000 trees were grown with other tuning parameters determined by cross-validation.  Similarly, XGBoost's hyperparameters were tuned through a cross-validation scheme.  For the DNN model, what we adopted was a revised DNN ensemble to deal with the unstable prediction challenge due to the random parameter initialization in the conventional DNN algorithm.  In the revised DNN model, two procedures were introduced to address the unstable prediction issue, i.e., bootstrap aggregating and filtering [50].  Our DNN model utilized an ensemble of 100 models, with 4 hidden layers, 50, 40, 30, 20 hidden nodes from the first to the last layer, respectively, employing a mini batch size 30.

Results

ICU Surgical Patients’ Basic Characteristics

In the cohort of 4,912 surgical patients, all were aged 18 or older, with a mean age of 65.2 years. Of these, 3,127 (63.7%) were male, and 1,785 (36.3%) were female. Among the patients, 2,551 (51.9%) underwent cardiac surgeries, 1,089 (22.2%) had general surgeries, while 527 (10.7%), 284 (5.8%), 248 (5.0%), 186 (3.8%), and 27 (0.5%) underwent neurologic, circulatory, thoracic, musculoskeletal, and plastic surgeries, respectively. A summary of the demographic characteristics for 4,912 surgical patients included in this study is presented in Table 1.

Table 1: Summary of Key Demographic Features.

Model Performance Comparison on Postoperative Opioid Usage Prediction

Under the permutation feature importance test framework, we identified the important features associated with average daily postoperative opioid consumption for each of machine learning models (DNN, RF, SVM, XGBoost).  The identified important features for each model were then included in the corresponding machine learning model for predicting average daily postoperative opioid consumption based on the testing data. The models are referred to as DNN, RF, SVM, and XGBoost, respectively, and they are interpretable since each of the input feature’s impact on the outcome can be expressively evaluated.  It is worth noting that when training the machine learning predictive models and identifying the important features for each machine learning model all the testing samples were withheld without leaking to influence model training and feature importance evaluation during model training and feature importance identification processes to avoid overfitting.  The results of the analysis are presented in Table 2.

Table 2: Performance Comparison for Daily Postoperative Opioid Consumption.

Table 2 reveals that in the reduced models where only the identified significant features (p-value < 0.05) were included for predicting average daily postoperative opioid consumption, the MSE and PCC along with their corresponding 95% confidence intervals (CI), were as follows: DNN (7889.23 [5751.744, 10026.716], 0.283 [0.165, 0.401]), SVM (8810.669 [6034.788,11586.550], 0.236 [0.152, 0.320]), RF (8214.330 [6180.240, 10248.420], 0.225 [0.115, 0.335]), and XGBoost (8681.910 [6054.787, 11309.033], 0.217 [0.144, 0.290]). DNN outperformed all other models in terms of all evaluation metrics considered.

Identified Important Features Associated with ICU Patient’s Postoperative Opioid Use

Using the permutation feature importance test framework, we identified the important features associated with postoperative opioid consumption for each machine learning model across each fold of 10-fold cross-validation. Figure 2 displays the heatmap of the frequency of identified important features for each machine learning method at a significance level of 0.05. It's evident from this figure that patient age and surgery type are consistently identified as significant factors for average daily postoperative opioid consumption across all 10 folds by all four machine learning methods.  Moreover, sex, serum calcium, serum potassium, diabetes, admission type, alcohol abuse, and cerebrovascular disease/stroke were identified as significant factors associated with average daily postoperative opioid consumption across all 10 folds by the DNN model.

Figure 2: Frequency Heatmap of Features.

Discussion

The management of opioids in ICU surgical patients during the early recovery period post-surgery poses significant clinical challenges, requiring accurate prediction and understanding of factors influencing opioid consumption for optimal pain management strategies. In this prognostic study, we comprehensively evaluated a range of commonly used machine learning models, including SVM, XGBoost, RF, and DNN, to construct and validate a robust predictive model for accurate postoperative opioid consumption in ICU surgical patients using EHR data from clinical practice. Among these models, the DNN model outperformed others by achieving the lowest mean squared error and the highest Pearson correlation coefficient. This underscores the effectiveness of the DNN model in capturing the intricate associations between clinical features and postoperative opioid consumption. Using a permutation feature importance test framework, the DNN model identified a total of 13 significant features associated with postoperative opioid consumption in every fold of the 10-fold cross-validation analysis, including age, surgery type, sex, serum CO2 levels, serum calcium, serum potassium, alcohol/drug abuse history, cerebrovascular disease, admit location, diabetes, and admit type among a total of 25 features. While many of these findings align with prior research, some are novel. For instance, studies have consistently shown a decrease in postoperative opioid prescription with increasing age [23,51,52]. Additionally, research has highlighted the physiological connection between opioid usage and vital signs such as serum potassium, CO2 level, and platelet count[53]. Notably, the Permutation Feature Importance Test (PermFIT) utilized in this study differs significantly from other existing feature selection methods like the Shapley Value-Based Method (SHAP)  [54]. While PermFIT not only assesses the importance score for each individual feature but also provides statistical inference, enabling the identification of statistically significant features, SHAP solely evaluates each feature’s importance score for relative importance ranking. This suggests that features identified via SHAP could be statistically insignificant, potentially misleading clinical decision-making. This study highlights the potential of using interpretable deep learning frameworks to enhance precision opioid prescription in ICU surgery. By identifying crucial influencing features, our approach offers a valuable tool for tailoring opioid prescriptions to the individual needs of surgical patients in ICUs, potentially improving patient outcomes and enhancing safety.

Limitation

This study was limited by the absence of certain crucial features, such as preoperative pain scores and surgery duration. These factors are important as they may significantly impact postoperative opioid consumption. Future research should aim to include these variables to provide a more comprehensive understanding of the determinants of opioid use in ICU surgical patients.

Conclusion

The study results underscore the effectiveness of the PermFIT-DNN model in flexibly identifying significant features for robust and accurate postoperative opioid consumption prediction, accompanied by solid statistical interpretations. This model not only enhances prediction accuracy using these identified important features but also provides valuable insights into the factors influencing opioid use in ICU surgical patients. These findings hold potential for informing the development of predictive tools to facilitate evidence-based clinical decision-making, particularly regarding personalized postoperative opioid prescription recommendations. However, the study’s limitations suggest that some important features related to postoperative opioid consumption, such as preoperative pain scores and surgery duration, may not have been captured. This highlights the need for more comprehensive investigations. Future research should aim to integrate a broader collection of postoperative opioid consumption data to train the DNN model within the PermFIT framework. By doing so, a more complete set of important features could be identified, further enhancing the prediction accuracy and ultimately improving patient outcomes in ICU settings.

Acknowledge:This study was partially supported by NIH (National Institutes of Health) R56 grant (1R56LM013784-01A1) and R01 grant (1R01LM014407).

References

1. Hall MJ, Schwartzman A, Zhang J, Liu X (2017) Ambulatory surgery data from hospitals and ambulatory surgery centers: United States, 2010. Natl Health Stat Report 102: 1-15.
2. Steiner CA, Karaca Z, Moore BJ, Imshaug MC and Pickens G (2006) Surgeries in hospital-based ambulatory surgery and hospital inpatient settings, 2014: Statistical Brief #223.
3. Wunsch H, Wijeysundera DN, Passarella MA, Neuman MD (2016) Opioids prescribed after low-risk surgical procedures in the United States, 2004-2012. JAMA 315: 1654-1657.
4. Chen Q, Chen E, Qian X (2021) A narrative review on perioperative pain management strategies in enhanced recovery pathways—the past, present and future. J Clin Med 10: 2568.
5. Brummett CM, Waljee JF, Goesling J, Moser S, Lin P (2017) New persistent opioid use after minor and major surgical procedures in US adults. JAMA Surg 152: e170504.
6. Rudd RA, Seth P, David F, Scholl L (2016) Increases in drug and opioid-involved overdose deaths — United States, 2010–2015. MMWR Morb Mortal Wkly Rep 65: 1445-1452.
7. NIH HEAL Initiative (2023) NIH HEAL Initiative | The Opioid Crisis.
8. Cdc.gov (2024) Products - Vital Statistics Rapid Release - Provisional Drug Overdose Data.
9. Rudd RA, Aleshire N, Zibbell JE, Gladden RM (2016) Increases in drug and opioid overdose deaths -- United States, 2000-2014. MMWR Morb Mortal Wkly Rep 64: 1378-1382.
10. Peterson AB, Gladden RM, Delcher C, Spies E, Williams AG (2016) Increases in fentanyl-related overdose deaths - Florida and Ohio, 2013-2015. MMWR Morb Mortal Wkly Rep 65: 844-849.
11. Racine M (2018) Chronic pain and suicide risk: A comprehensive review. Prog Neuropsychopharmacol Biol Psychiatry 87: 269-280.
12. Kuo A, Wyse BD, Meutermans W, Smith MT (2014) In vivo profiling of seven common opioids for antinociception, constipation and respiratory depression: No two opioids have the same profile. Br J Pharmacol 171: 1580-1590.
13. Hagle ME, Lehr VT, Brubakken K, Shippee A (2004) Respiratory depression in adult patients with intravenous patient-controlled analgesia. Orthop Nurs 23: 18-27.
14. Baldo BA (2021) Toxicities of opioid analgesics: Respiratory depression, histamine release, hemodynamic changes, hypersensitivity, serotonin toxicity. Arch Toxicol 95: 2627-2642.
15. Pasero C, McCaffery M (2002) Monitoring sedation: It's the key to preventing opioid-induced respiratory depression. American Journal of Nursing 102: 67-69.
16. Dowell D, Ragan KR, Jones CM, Baldwin GT, Chou R (2022) CDC clinical practice guideline for prescribing opioids for pain — United States, 2022. MMWR Recomm Rep 71: 1-95.
17. CDC (2022) Summary of the 2022 Clinical Practice Guideline for Prescribing Opioids for Pain | Opioids | CDC.
18. Abdulla A, Adams N, Bone M, Elliott AM, Gaffin J (2013) Guidance on the management of pain in older people. Age Ageing 42 Suppl 1: i1-i57.
19. Chen D, Li X, Chen Y, Zeng H, Liu J, Li Q (2022) Opioid dose, pain, and recovery following abdominal surgery: A retrospective cohort study. J Clin Med 11: 7320.
20. Shanthanna H, Ladha KS, Joshi GP, Kehlet H (2021) Perioperative opioid administration: A critical review of opioid-free versus opioid-sparing approaches. Anesthesiology 134: 645-659.
21. Jivraj NK, Raghavji F, Ladha KS, Wijeysundera DN, Ladha KS (2020) Persistent postoperative opioid use: A systematic literature search of definitions and population-based cohort study. Anesthesiology 132: 1528-1539.
22. Bartley EJ, Fillingim RB (2013) Sex differences in pain: A brief review of clinical and experimental findings. Br J Anaesth 111: 52-58.
23. Bethell J, Neuman MD, Bateman BT, Hill AD, Ladha KS (2020) Age and postoperative opioid prescriptions: A population-based cohort study of opioid-naïve adults. Pharmacoepidemiol Drug Saf 29: 504-509.
24. Thiels CA, Anderson SS, Ubl DS, Hanson KT, Bergquist WJ (2017) Wide variation and overprescription of opioids after elective surgery. Ann Surg 266: 564-573.
25. Ip HYV, Abrishami A, Peng PWH, Wong J, Chung F (2009) Predictors of postoperative pain and analgesic consumption: A qualitative systematic review. Anesthesiology 111: 657-677.
26. Small C, Laycock H (2020) Acute postoperative pain management. Br J Surg 107: e70-e80.
27. Pirie K, Traer E, Finniss D, Myles PS, Riedel B (2022) Current approaches to acute postoperative pain management after major abdominal surgery: A narrative review and future directions. Br. J. Anaesth 129: 378-393.
28. Wong M, Vogell A, Wright K, Isaacson K, Loring M (2019) Opioid use after laparoscopic hysterectomy: prescriptions, patient use, and a predictive calculator. Am J Obstet Gynecol 220: 259.e1-259.e11.
29. Santosa KB, Wang CS, Hu H-M, Brummett CM, Englesbe MJ, Waljee JF (2020) Surgeon experience and opioid prescribing. J Am Coll Surg 220: 823-827.
30. Janakiram C, Fontelo P, Huser V, Chalmers NI, Lopez Mitnik G, et al. (2019) Opioid prescriptions for acute and chronic pain management among Medicaid beneficiaries. Am J Prev Med 57: 365-373.
31. Behman R, Cleary S, McHardy P, Kiss A, Sawyer J (2019) Predictors of post-operative pain and opioid consumption in patients undergoing liver surgery. World J Surg 43: 2579-2586.
32. Zhang KK, Blum KM, Chu JJ, Zewdu A, Janse S (2023) A personalized opioid prescription model for predicting postoperative discharge opioid needs. Plast Reconstr Surg 151: 450-460.
33. Sandhu S, Lin AL, Brajer N, Sperling J, Ratliff W (2020) Integrating a machine learning system into clinical workflows: Qualitative study. J Med Internet Res 22: e22421.
34. Zorc JJ, Chamberlain JM, Bajaj L (2019) Machine learning at the clinical bedside—The ghost in the machine. JAMA Pediatr 173: 622-624.
35. Hastie T, Tibshirani R, Friedman J (2009) The Elements of Statistical learning, Second Edition : Data mining, inference, and Prediction. 2nd ed. [online] New York: Springer.
36. Drucker H, Burges CJC, Kaufman L, Smola A, Vapnik V (1996) Support vector regression machines. Proceedings of the 9th International Conference on Neural Information Processing Systems (NIPS'96) MIT Press 1996: 155-161.
37. Cortes C, Vapnik V (1995) Support-vector networks. Machine Learning 20: 273-297.
38. Chen T, Guestrin C (2016) XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16) 2016: 785-794.
39. Breiman L (2001) Random forests. Machine Learning 45: 5-32.
40. Yali A, Donald G (1997) Shape quantization and recognition with randomized trees. Neural Comput 9: 1545-1588.
41. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521: 436-444.
42. Bengio Y (2009) Learning deep architectures for AI. Foundations and Trends in Machine Learning 2: 1-127.
43. Amann J, Blasimme A, Vayena E, Frey D, Madai VI (2020) Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med Inform Decis Mak 20: 310.
44. Stiglic G, Kocbek P, Fijacko N, Zitnik M, Verbert K (2020) Interpretability of machine learning-based prediction models in healthcare. WIREs Data Mining Knowl Discov 2020: e1379.
45. Zhou Y, Kantarcioglu M (2020) On transparency of machine learning models: A position paper. AI for Social Good Workshop.
46. Mi X, Zou B, Zou F, Hu J (2021) Permutation-based identification of important biomarkers for complex diseases via machine learning models. Nat Commun 12: 3008.
47. Johnson AE, Pollard TJ, Shen L, Lehman LW, Feng M, et al. (2016) MIMIC-III, a freely accessible critical care database. Sci Data 3:160035.
48. Ramos-Matos CF, Lopez-Ojeda W, Bistas KG (2023) Fentanyl.
49. Dowell D, Ragan KR, Jones CM, Baldwin GT, Chou R (2022) CDC clinical practice guideline for prescribing opioids for pain — United States, 2022. MMWR Recomm Rep 71: 1-95.
50. Mi X, Tighe P, Zou F, Zou B (2021) A deep learning semiparametric regression for adjusting complex confounding structures. Ann Appl Stat 15: 1086-1100.
51. Shanahan CW, Reding O, Holmdahl I, Keosaian J, Xuan Z (2021) Opioid analgesic use after ambulatory surgery: A descriptive prospective cohort study of factors associated with quantities prescribed and consumed. BMJ Open 11: e047928.
52. Sun EC, Darnall BD, Baker LC, Mackey S (2016) Incidence of and risk factors for chronic opioid use among opioid-naive patients in the postoperative period. JAMA Intern Med 176: 1286-1293.
53. Gruba SM, Francis DH, Meyer AF, Spanolios E, He J (2022) Characterization of the presence and function of platelet opioid receptors. ACS Meas Sci Au 2: 4-13.
54. Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 2017: 4765-4774.