Inter-observer agreement in classifying anesthetic deaths in cats and dogs

Abstract

Determining the cause of death in studies assessing mortality during small animal anesthesia poses challenges due to varying definitions of anesthetic death, limited information, and differences in evaluators’ interpretations. This study aims to establish the interobserver agreement in classifying the cause of death in anesthetized animals. The observational, retrospective, multicenter study analyzed 432 deaths (83 cats and 349 dogs). Data were collected from a database of 55,022 anesthetized dogs and 14,962 anesthetized cats, created to investigate anesthetic-related mortality in these species. Three highly qualified veterinary anesthesiologists independently assessed whether the deaths were related to anesthesia, using their professional judgment. Data were collected from questionnaires that included the animal’s signalment, reason for anesthesia, ASA status, drugs, anesthetic procedures, and comments from the submitting veterinarian. Light’s Kappa and the intraclass correlation coefficient (ICC) measured interrater agreement among the three evaluators, while Cohen’s Kappa assessed interrater reliability between pairs of observers (p < 0.05). Evaluators A, B, and C classified 296/432 (68.5%), 264/432 (61.1%), and 54/432 (12.5%) of the cases as anesthesia-related deaths, respectively. Agreement among the three evaluators was 128/432 (29.6%) [Light’s Kappa: 0.17, p = 0.00026; ICC: 0.06, p-value = 0.0167]. The three evaluators agreed on classifying a death as anesthetic-related in 14.1% of cases (50 out of 354 cases where at least one evaluator classified the death as anesthetic-related). Similarly, for non-anesthetic-related deaths, the three evaluators reached an agreement in 20.4% of cases (78 out of 382 cases where at least one evaluator classified the death as non-anesthetic-related). Overall, agreement between two out of three evaluators was 304/432 (70.4%). Evaluators A and B had a 65.7% agreement [Cohen’s Kappa: 0.25, p < 0.00001], A and C had a 46.6% agreement [Cohen’s Kappa: 0.10, p < 0.00001], and B and C had a 50.9% agreement [Cohen’s Kappa: 0.16, p < 0.00001]. In conclusion, the evaluators’ agreement was weak, highlighting the need for a consensus on defining anesthetic mortality in dogs and cats.

Introduction

Anaesthesia is a state of reversible unconsciousness encompassing immobility, amnesia, hypnosis, analgesia, and muscle relaxation [1]. This process significantly alters the patient’s physiological condition and can precipitate a range of anesthetic complications [2], including death in cats, dogs [2,3,4,5,6,7,8,9,10,11,12,13] and humans [14,15,16].

Anesthetic mortality, defined as the percentage of patients dying while undergoing anesthesia, is a critical benchmark for evaluating the safety of anesthetic procedures [7,8,9, 12, 13]. Reported anesthesia-related mortality rates in dogs and cats range from 0.17 to 1.35% nd 0.24%to 5.80% [3,4,5,6,7,8,9,10,11,12,13, 17, 18] This rat is substantially higher than that reported for humans, which is approximately 0.00001%–-0.00002% in developed ountries [14,15,16].Identifying factors associated with anesthetic mortality may allow for prevention and, thus, reduction in mortality in veterinary medicine [7,8,9, 12, 13, 18].

A consistent definition is essential to understand anesthesia-related deaths fully. Current studies employ varying definitions and perioperative time frames, complicating trend analysis in anesthesia safety [12, 13, 19]. The first study on anesthetic deaths in veterinary medicine classified all deaths from induction to recovery as anesthesia-related [3]. Later, multi-center research categorized perioperative deaths based on patient risk, attributing deaths in low-risk patients (ASA I and II) to anesthesia unless a clear surgical cause was present, while all deaths in high-risk patients (ASA ≥ III) were considered anesthetic-related [4]. Hosgood and Scholl expanded the definition to include deaths within 24 h post-anesthesia regardless of cause [5, 6]. Some studies further refined this to deaths from premedication to 24 h post-procedure, excluding euthanasia [20, 21].

Mortality rates have also considered deaths related to both sedation and anesthesia [8, 10, 18]. Itami et al. (2017) defined anesthetic death as occurring from premedication to 48 h after extubation, excluding surgical errors and euthanasia-related deaths [17]. Another study defined anesthetic/sedative-related death within 48 h or two weeks of anesthesia if the agents could not be excluded as contributing factors [18]. Recent studies in cats [12] and dogs [13] defined anesthesia-related death as directly or partially attributed to anesthesia occurring from premedication to 48 h after extubation.

In human medicine, anesthesia-related deaths have been categorized as (a) deaths during anesthesia, (b) deaths without regaining consciousness after anesthesia, and (c) deaths resulting from anesthesia after regaining consciousness [22]. This categorization illustrates the complexity of identifying anesthesia’s role in patient mortality. Anesthesia-related mortality generally refers to deaths occurring during or after surgery under anesthesia. However, establishing a direct causal link between anesthesia and patient death is defiant, complicating the measurement of anesthesia-associated mortality and the assessment of its impact on outcomes [23]. To address these challenges, it has been suggested that the definition of anesthesia-related mortality should include only those deaths directly resulting from anesthetic procedures [23]. An editorial emphasized the need for a standardized definition to ensure clarity and uniformity [24]. The lack of consensus on this definition highlights the ongoing efforts to enhance anesthesia safety [25].

The investigation of peri-anesthetic mortality has attracted considerable attention. However, there is significant variability in defining peri-anesthetic mortality, and there are no tests specific to diagnosis that can confirm whether a death was caused by anesthesia or not. Death attribution to anesthesia is based on expert opinion [8, 12, 13, 18], making it complex. We hypothesize that the lack of a standardized definition of peri-anesthetic death will lead to significant variability in the classification of the cause of death among different observers. Hence, our study aims to assess the inter-observer agreement classifying the cause of death in anesthetized cats and dogs.

Materials and methods

This observational, retrospective, multicenter study received ethical approval from the Ethics Committee of the Universidad CEU Cardenal Herrera, Spain (CEEA 22/07). This study analyzed 432 deaths —83 cats and 349 dogs— extracted from a database of 55,022 anesthetized dogs and 14,962 anesthetized cats created for a study investigating anesthetic-related mortality in these species. The studied animals were anesthetized and died from premedication until 48 h after extubation. The clinician-in-charge of the case documented all relevant information in a form (Supplementary Material 1). The collected data included signalment, patient history, medical history, pre-anesthetic assessment findings, ASA status, the purpose of anesthesia, and details of drugs administered during the procedure (including dosage, method of administration, and timing). The study also recorded the locoregional techniques performed, anesthesia monitoring, intra- and post-anesthetic complications, duration of anesthesia, and time of death. Information was provided regarding whether the procedure performed was scheduled elective, unscheduled elective, or urgent. It also included whether the procedure occurred during regular working hours or out-of-hours. Finally, comments were provided.

After anonymization, the information was presented to Three highly qualified veterinary anesthesiologists who independently assessed whether the deaths were connected to anesthesia based on their professional judgment. Two evaluators were board-certified, one by the European College of Veterinary Anaesthesia and Analgesia (ECVAA), and the other held a doctorate and an honorary diploma from the American College of Veterinary Anesthesia and Analgesia (ACVAA). The third evaluator also had a doctorate in veterinary anesthesia and was ECVAA-eligible. They were randomly assigned the letters A, B, and C. Using their professional judgment without any suggestion or advice, the evaluators were asked to classify these cases into two categories individually: (a) anesthetic-related death (if the death could be directly or partially attributed to anesthesia) and (b) non-anesthetic-related death (if the death resulted from surgical complications or disease progression during the study period or other reasons not related to anesthesia).

The statistical analysis used R 4.4.1, a language and environment for statistical computing and graphics. Descriptive statistics included the number of cases related to anesthesia for each anesthesiologist and the percentage of agreement between two or all three of them. The results were analyzed using inferential statistics. Light’s kappa and the intraclass correlation coefficient (ICC) were used to evaluate the inter-rater agreement between the three consultants for categorizing anesthetic-related deaths. In contrast, Cohen’s kappa was used to assess the reliability of the two evaluators. Chi-square tests of independence were conducted for some variables of interest (such as species, ASA, timetable, type of anesthesia, reason for anesthesia, duration, moment of death, hospitalization, level of monitoring, scheduling, and emergency) to examine potential associations between them and the agreement. In all analyses, differences were considered statistically significant when p < 0.05.

Results

The three evaluators (A, B, and C) reviewed 432 cases. Evaluator A classified 296 cases (68.5%), Evaluator B classified 264 cases (61.1%), and Evaluator C classified 54 cases (12.5%) as anesthesia-related deaths, respectively.

The overall agreement among the three Evaluators was 128 cases (29.6%). Light’s kappa yielded a value of 0.171 (p-value = 0.00026), suggesting low concordance among the evaluators and indicating that while there is some agreement beyond what would be expected by chance (which would be 25%), the extent of this agreement is modest. The ICC was calculated at 0.06 [IC: 0.005 to 0.119], reflecting a minimal proportion of variance attributable to differences between evaluators (p-value = 0.0167), highlighting that despite the low ICC, there are statistically significant differences in evaluations, underscoring the limited level of agreement among evaluators. These results indicate that while the evaluators demonstrate some consistency, the agreement on the anesthetic-related death categorization is relatively low.

The three evaluators agreed on classifying a death as anesthetic-related in 14.1% of cases (50 out of 354 cases where at least one evaluator classified the death as anesthetic-related). Similarly, for non-anesthetic-related deaths, the three evaluators reached an agreement in 20.4% of cases (78 out of 382 cases where at least one evaluator classified the death as non-anesthetic-related). Additionally, pairwise agreements were observed. Overall, agreement between two out of three evaluators was 304/432 (70.4%). Evaluators A and B had a 65,7% agreement (46.8% for anesthetic and 20.4% for non-anesthetic deaths [Cohen’s Kappa: 0.25, p < 0.00001], A and C had a 46.6% agreement (14.4% for anesthetic and 34.8% for non-anesthetic deaths [Cohen’s Kappa: 0.10, p < 0.00001], and B and C had a 50.9% agreement (15.0% for anesthetic and 43.7% for non-anesthetic [Cohen’s Kappa: 0.16, p < 0.00001]. Figures 1 and 2 show Venn’s diagrams of the inter-observer agreement of the three evaluators on diagnosing the cause of death related to anesthesia or not, respectively.

Fig. 1

Venn diagram of inter-observer agreement of the three evaluators (A, B, and C) on classifying the cause of death as anesthetic-related

Fig. 2

Venn diagram of inter-observer agreement of the three evaluators (A, B, and C) on diagnosing the cause of death unrelated to anesthetic reasons

No differences were noted comparing the proportion of agreement (or disagreement) and other variables of interest, such as species (p = 0.7701), ASA (p = 0.0979), timetable (p = 0.3607), type of anesthesia (p = 0.8473), reason for anesthesia (p = 0.5358), duration (p = 0.4363), moment of death (p = 0.1923), hospitalization (p = 0.2279), level of monitoring (p = 0.5192), scheduling (p = 0.4108) or emergency (p = 1). Table 1 shows the evaluators’ comprehensive agreement and disagreement according to the studied factors.

Table 1 Univariate analysis of the agreement and disagreement between the observers according to the studied factors. N: number of cases. %: percentage of cases

Discussion

This study revealed significant inter-observer variability in categorizing anesthesia-related mortality in cats and dogs. Our findings confirmed the hypothesis that anesthetists are inconsistent in classifying the cause of death, indicating substantial biases and discrepancies in such classifications.

In this study, three experienced anesthetists independently evaluated anesthesia-related mortality as either ‘anesthesia-related’ or ‘non-anesthesia related,’ resulting in a potential discrepancy rate of 33.3% and a maximum potential agreement of 100%. The observed agreement rate for classifying anesthetic-related deaths was only 14.1%, indicating significant disparity among the evaluators. This low agreement rate suggests considerable variability in the interpretation of criteria for anesthetic mortality classification, likely due to subjective interpretations, varying experiences, and inherent ambiguities in the criteria.

There is a clear interest in identifying and classifying the cause of death in the medical field. Previous studies have attempted to find objective methods to determine the cause of perianesthetic mortality, but this has generally not been successful. Post-mortem toxicological analyses in animals, for example, offer limited utility in assessing possible anesthetic drug overdose or adverse effects [26]. A study of postmortem injuries in domestic animals revealed that surgical or anesthesia-associated complications were identified in only 5% o cases [27].

Therefore, to the authors´ knowledge, there is no objective way to identify death as an anesthetic cause. Death classification as anesthetic-related relies heavily on specialist evaluations [7,8,9, 12, 13, 18]. A clear definition of peri-anesthetic mortality is critical yet challenging, as it is difficult to ascertain if patients died solely from anesthetic causes—i.e. if they would have survived without anesthesia. Furthermore, deaths often have multifactorial causes [28], making it even more challenging to classify them as solely attributable to anesthesia. Therefore, the question should perhaps not be whether anesthesia was the direct cause of death but rather to determine and assess its degree of influence on the fatal outcome. To address these discrepancies and improve reliability and validity in future studies, it is essential to implement more rigorous training protocols for evaluators, refine classification criteria, and introduce objective measures to reduce subjectivity. More precise guidelines and additional training could enhance consensus among evaluators. The variability in classifying anesthetic-related deaths poses a significant challenge for comparing different studies on anesthesia-related mortality. This inconsistency complicates identifying risk and protective factors to reduce such mortality. Further research is needed to understand the reasons for these differences in classification and how to address them.

Human medicine has already recognized the need for consensus on the definition of peri-anesthetic mortality. A clear definition of anesthesia-related critical events could improve clinical practice by enhancing planning, resource allocation, and preoperative discussions with patients [24]. Unfortunately, this consensus has not yet been reached in human or veterinary medicine.

This problem of subjectivity is not unique to the classification of the cause of perioperative deaths. For example, the ASA classification, initially developed for human anesthesia [29] and subsequently adapted for veterinary use, although proven useful [30], is also susceptible to subjective interpretations. Studies have shown that agreement between human anesthesiologists in assigning ASA scores is fair to moderate [31]. The same problem is likely to be encountered in veterinary medicine. In our study, the evaluators were given subjective information, such as the patient’s ASA status, which could lead to biased interpretations when classifying a death as anesthetic or non-anesthetic. However, assessment disparities persist despite providing the same information to all three evaluators. The future may lie in developing objective tools such as the Combined Horse Anesthetic Risk Identification and Optimization Tool (CHARIOT), which aims to increase objectivity in assessing anesthetic risk in equine patients [32, 33]. In our case, we emphasize that developing an objective tool for classifying the cause of death requires first establishing a consensus on the definition of perianesthetic mortality.

A clear, standardized definition of anesthetic-related death is also critical for Morbidity and Mortality conferences (M&MCs). These meetings are essential in healthcare organizations, providing a framework to review patient care processes, identify safety issues, and enhance quality improvement [34]. M&MCs are pivotal in clinical settings as they facilitate the systematic review of adverse events and complications to improve patient care [35]. Without a consistent and precise definition of anesthetic-related death, the discussions and conclusions drawn in such conferences can be compromised by subjective interpretations and inconsistent criteria. This may lead to varied conclusions about the causes of death, hindering the ability to identify patterns and implement effective strategies for risk reduction.

An essential consideration in this study is the potential for case selection bias. All cases included in the analysis provided sufficient information for classification, and there were no instances where evaluators could not categorize a death based on the available data. Cases were selected according to predefined criteria, ensuring only submissions with comprehensive documentation were included. While this approach mitigated issues related to incomplete data, it may have inadvertently introduced selection bias, as cases with less thorough or incomplete submissions were excluded. This limitation highlights the importance of ensuring that future studies strive for uniformity in data collection across all cases to minimize potential biases and enhance the generalizability of findings.

On the other hand, while standardized evaluation and classification guidelines are crucial for improving inter-observer agreement and minimizing subjectivity, their effectiveness largely depends on the uniform application of comprehensive anesthetic protocols and monitoring practices. Without appropriate monitoring (e.g., ECG and blood pressure), or if the recorded data are misinterpreted, even the most stringent evaluation guidelines may fail to produce reliable conclusions. To enhance accuracy, standardized protocols for both monitoring and evaluation are essential to provide a solid framework for determining causes of death.

The challenge of understanding anesthesia-related mortality remains unresolved. Key questions include whether anesthesia-related death should be defined as directly caused by anesthesia or influenced by it. Reaching a consensus on this definition in veterinary medicine is imperative, and developing tools to classify deaths attributable to anesthesia would be a significant step forward.

In summary, the absence of a universally accepted definition of peri-anesthetic death hinders objective assessment of anesthesia-related mortality rates. This ambiguity complicates the exploration of underlying causes and risk factors, making reduction efforts difficult. Our study highlights the need to develop a standardized cause-of-death classification system to eliminate subjectivity and improve clinical decision-making processes among veterinary anesthetists. Such a consensus is precious for M&M conferences, where consistent and accurate classifications can lead to better understanding and improvements in clinical practice.

Conclusion

There is a lack of consensus in identifying and classifying anesthesia-related deaths. Defining anesthesia mortality clearly could lead to the creation of a tool for objectively categorizing the cause of death. Such a tool could improve clinical decision-making, identify anesthesia-related risk and protective factors, and enhance discussions at morbidity and mortality conferences.