As the spread of antibiotic-resistant infections has increased in recent years, a global health crisis has emerged with severe health and economic implications [1]. A study from the UN Interagency Coordinating Group on Antimicrobial Resistance cautioned that if no new major developments are made by 2050, mortality could rise to 10 million deaths globally each year [2]. A critical threat is posed in healthcare facilities from rising drug-resistant infections, as effective treatments are lacking [3].

A recent study [4] presented the resistance levels of multidrug-resistant isolates between the intensive care unit and other hospital wards in a public tertiary hospital in Greece. Feretzakis et al. [5] proposed a methodology that enables clinicians to select the most appropriate antibiotic therapy based on statistically significant sensitivity results from data available in the laboratory information system (LIS). The use of unit-specific local antibiograms within hospitals is highly recommended by the Infectious Disease Society of America (IDSA) as a principal guideline for the prescription of effective empiric treatment [6].

Artificial intelligence (AI) tools are increasingly applied in healthcare, potentially changing many aspects of patient care as well as administrative processes within hospitals. Several researchers suggest that AI can work in key healthcare activities, such as diagnosis and treatment, with equal or even greater accuracy than clinicians. A recent study [7] reported that AI outperformed experts by identifying cancers that radiologists missed in the images while ignoring features they falsely identified as possible tumors. The potential for AI in healthcare has been described in detail [8]. A recent review article [9] presented the opportunities offered by automated machine learning (AutoML) platforms for healthcare.

The current literature shows great interest in implementing machine learning (ML) techniques as clinical decision support tools for the prediction of antimicrobial resistance (AMR) [10–12] and the selection of appropriate empirical antibiotic treatment. Still, ML techniques are not widely implemented in clinical practice in this particular domain since clinicians take into account several patient-specific factors to choose an empirical therapy. Predictive models for antibiotic susceptibility can be an additional tool for decision support regarding early empirical therapy [13].

In this study, we assessed the effectiveness of AutoML-trained models to predict AMR based only on data available in the LIS of the clinical microbiology laboratory, such as the type of sample, the Gram stain, and the antibiotic susceptibility results together with simple patient demographics (age/sex). Age and sex have been reported in research studies [14,15] as factors influencing AMR.

We retrospectively analyzed the antimicrobial susceptibility data of the biopathological laboratory from 499 patients admitted to the internal medicine wards of a public hospital in Greece from January until December 2018. This study was approved by the Institutional Review Board of Sismanogleio General Hospital (No. 6682/2020). The dataset consisted of 11,496 instances and contained the attributes of sex (binary), age (numerical), sample type (categorical), Gram stain (positive or negative; binary), 44 antimicrobial substances (categorical), and the antibiotic susceptibility result (sensitive or resistant; binary). The different types of clinical samples that were taken into consideration for the antibiotic susceptibility analysis, together with simple summary statistics of the dataset, are presented in Table 1.

AutoML automates the application of various ML techniques and enables researchers to develop large-scale and effective predictive models. Traditional ML model development is resource-intensive, as it needs significant domain knowledge and time to build and compare the performance of many models. For our experiments, we used the AutoML of the Microsoft Azure ML platform, and the ML algorithms during the automation and tuning process have been fully described [16]. In AutoML experiments, automatic scaling and normalization techniques are applied to all data by default. Since the purpose of our research paper is to present an easy-to-apply procedure that can be communicated to and even used by non-technical experts, we kept all the default parameters and avoided applying custom settings at all stages, including the feature selection process. Our proposed approach is summarized in Figure 1.

For the same reason, we chose to use a 10-fold cross-validation approach to evaluate the performance of the deduced models instead of using custom settings to split the data into training and testing sets. Cross-validation is an ML technique that uses all available instances for training and testing. It mimics the use of training and test sets by repeatedly training the algorithm K times with a fraction 1/K of training examples left out for testing purposes [17].

Due to the imbalance of our dataset, we also examined the performance of AutoML algorithms by applying an oversampling method since we wanted to keep all the intrinsic value of our dataset and avoided dropping any possible valuable instances by using an undersampling method. Synthetic minority oversampling technique (SMOTE) is a statistical method for uniformly increasing the number of cases in a dataset in order to make it balanced. The new instances are not just duplicates of existing minority instances. Instead, this method takes feature space samples for each target class and its nearest neighbors. After that, new instances are produced that combine features of the target case with those from its neighbors [18].

The performance metrics [19] we used in our analysis are briefly presented below:

The results of the four top-performing techniques with 10-fold cross-validation are shown in Table 2.

The best overall results were achieved by a stack ensemble technique with an AUCW metric of 0.822, an APSW of 0.834, an F1W of 0.761, and an accuracy of 0.770. The feature importance was 0.660 for the antibiotics, 0.348 for sex, 0.305 for age, 0.294 for the type of sample, and 0.112 for the Gram stain. Feature importance values vary between zero and one, with higher values indicating a stronger association with predictions. The sensitivity (recall) of the best model was 0.539, and the specificity (precision) was 0.896.

Ensemble learning [21,22] is an ML approach where multiple models are trained to solve the same problem. Stacking is an ensemble ML technique that combines multiple classifications via a meta-classifier. The default meta-classifier of the stack ensemble algorithm in Microsoft Azure AutoML for classification tasks is logistic regression. Decision-making based on the feedback of many experts is a common practice that serves as the foundation of human civilization. In recent decades, AI researchers have studied schemes that share a joint decision-making mechanism of this type. These schemes are commonly called ensemble learning and tend to reduce the variance between classifiers and improve the decision-making system’s robustness and accuracy. In practical terms, we also seek advice from various experts when we make a decision that has significant implications. For example, we consult with many physicians before agreeing to undergo a major medical procedure [22].

As shown in Figure 2, we can visualize the confusion matrix for our best-performing model (stack ensemble). A confusion matrix is a n × n matrix used to describe the performance of a classifier, where n is the number of classes. Each row in a confusion matrix represents the instances of the actual class in the dataset, and each column represents the instances of the class that was predicted by the classification model [18].

None of the metrics that are presented above can be a one-off solution for the model’s evaluation, but all of them can help healthcare professionals to get a better picture of the overall performance of the classification models through the targeted guidance from data scientists.

It is worth noting at this point that a portion of the same dataset was processed in previous research [23], where five AI algorithms were evaluated. Those results suggested that the multilayer perceptron classifier could serve as a suitable model, taking into account evaluations using both the F-measure (0.721) and the area under the ROC curve (0.746). The corresponding results of the best model (stack ensemble) in this research are an F1 score-weighted (F1W) of 0.761 and an area under the curve-weighted (AUCW) of 0.822. These results are consistent with previous findings, and the results improved because using the AutoML platform enabled us to test and compare several models with different configurations in a time-efficient manner.

The results presented above are based on raw data without considering the balance of the target binary class (i.e., in our case, antimicrobial susceptibility). In the examined dataset, the class attribute (antimicrobial susceptibility) contained 66.68% positive cases (sensitive to a specific antibiotic) and 33.32% negative cases (resistant to a particular antibiotic).

In previous articles [24–26], researchers investigated the task of monitoring and detecting hospital-acquired infections, a topic related to AMR. In these studies, a significant issue was class imbalance, a problem that can be noticed in many classification tasks in the medical domain. This means that, in general, the prevalence of one class in the examined dataset is much higher than that of another one. However, most ML techniques provided better generalization when the number of samples is similar for both classes. In the literature, several methods have been suggested [27] to solve the issue of imbalance. In this study, we utilized an oversampling approach that aims to increase the number of minority samples according to the majority class.

The decision to start antibiotic therapy is faced in daily practice by clinicians and depends on complex clinical patient data and local epidemiology factors, as well as the information available at the time; the widespread dissemination of AMR further complicates empiric therapy decisions and raises the risk of therapy failure. Timely and appropriate initial therapy is of the utmost importance for sepsis outcomes. Molecular AMR diagnostics significantly reduce the time to achieve results compared to classical phenotypic tests, but aside from the high setup costs, the need for technical infrastructure, and staff training in bioinformatics, their major drawback is that they can detect only known resistance genes or mutations [29].

ML techniques, such as whole-genome sequencing, seem to be useful in AMR surveillance projects; however, the existing evidence does not support their use in guiding clinical decision-making for most bacterial species, according to the European Committee on Antimicrobial Susceptibility Testing [30].

The ML-based methodology proposed in this paper could empower physicians in decision-making while anticipating definitive results from the clinical microbiology laboratory on specific pathogen identification and antibiotic susceptibility testing, even in limited-resource settings. First, early recognition of patients at a high risk of being colonized or infected by strains resistant to one or more antibiotic classes leads to crucial patient and hospital ecosystem knowledge and subsequent improvement in healthcare resources management. More importantly, such systems may serve as a useful clinical decision support tool for physicians in selecting the appropriate empirical therapy. Thus, patient-tailored therapy can limit antibiotic misuse and, over time, reduce the prevalence of antibiotic-resistant bacteria.

Additionally, the proposed methodology is consistent with the practices of patient cohorting (placing patients who have been exposed to or infected with the same pathogen in the same inpatient room) or staff cohorting (assigning specific healthcare providers to care only for patients/residents known to be colonized or infected), which constitute an effective surveillance measure for multidrug-resistant infections that may prevent inadvertent patient-to-patient dissemination.

In this research, the clinical data of patients, such as the source of infection acquisition and the presence of active infection or colonization, were not included, as these were not readily accessible through the central hospital information system. Of course, if the antimicrobial susceptibility datasets also included the patient’s clinical details, the efficiency of the techniques that we present in this study could be considerably improved, even allowing for any final decision to be more explainable. Still, any inclusion of such information must incur the cost of retrieving the relevant data. This may involve a variety of hospital departments, thereby elevating the costs of communication and complicating the need to align protocols across different departments. After paying those costs, one would also need to review the extent to which the additional information gained (e.g., improved accuracy metrics) can be effectively incorporated into the practice of the hospital physicians, who may need to rethink the way they review their decisions in the light of confirming or opposing recommendations from a decision support system. This relates to the actual attitudes of physicians, who may need to consider modifying their practices, and the extent to which such considerations may also be triggered by a change in protocols or the extent to which such recommendations can be made within the prescribed time limits. All in all, we consider this study as a point in the spectrum of cost-effectiveness investigations that ML techniques are bound to trigger in the healthcare domain.

This study evaluated the results of applying AutoML of the Microsoft Azure platform to two internal medicine departments’ antimicrobial susceptibility datasets. In this article, we propose the use of AutoML as a decision tool for physicians since it can be more readily applied even by non-experts (e.g., a data scientist may be needed for a full-blown investigation, but a physician can gain some insight with a relatively smooth learning curve) and, as we showed, the deduced models have good performance.

AutoML platforms can be a beneficial tool for healthcare professionals with limited knowledge of the ML domain that can offer fast and reliable results. Ideally, data scientists’ participation could be important, especially in the stages of data pre-processing, by drawing actionable insights from the data, feature selection, and finally, evaluating the results.

Despite certain limitations of the study, our primary goal was to create an inexpensive ancillary tool to help clinicians rapidly identify patients carrying antibiotic-resistant strains and guide appropriate antibiotic treatment with greater confidence. In future work, dataset enhancement with clinical attributes will probably improve the AutoML algorithms’ performance.