Assessing quality in inpatient psychiatry using outcome scales from the interRAI-Mental Health

Modern discussions of hospital care quality have evolved from a conceptual framework proposed by Avedis Donabedian in a series of influential works published in the 1980s.^[1,2,3] According to Donabedian, the quality of care is proportional to the improvement in health status achieved by medical intervention, a definition that appears to have endured for the past 35 years.^[4] The model conceives of hospital care as comprising three components: Structure, Process, and Outcomes. Structure includes the hospital’s infrastructure (physical plant and systems, equipment, budgetary resources, and hospital staff). Process includes all diagnostic, surgical, medical, and nursing procedures, as well as policies, procedures, and best practices. The third component of the model is patient outcome. Health status is a metric that quantifies a patient’s health at a given point in time. If a patient’s health status is assessed prior to the provision of care (T1) and after the provision of care (T2), the difference reflects the improvement in health due to treatment. According to the model, in high-quality clinical programs, improvements in structure and process lead to better outcomes. We refer to T1vT2 differences as treatment effects. We recognize that demonstrating treatment efficacy requires appropriate experimental controls, such as those used in Randomized Controlled Trials, which are not included in Quality Assessment.

A recent integrative review of peer-reviewed and industry-implemented Balanced Scorecards (BSCs)^[5] described their use in quality improvement programs across mental health settings worldwide. While hospitals have had no difficulty identifying critical structural and process elements for inclusion in their BSCs, they have struggled to identify meaningful outcomes. Outcome measurement in psychiatry has recently received more serious attention in the literature.^[6] An important distinction is made between patient-reported and clinician-reported outcome measures. Most BSC outcome entries have been patient-reported,^[5] emphasizing patients’ and families’ subjective experiences. The research host hospital’s BSC contained multiple patient-reported measures. Additionally, clinician-reported outcomes have been limited in scope (e.g., % of patients with improved positive symptoms or changes in scores on traditional measures within a diagnostic category, such as depression). What has been lacking is an objectively quantified, clinician-scored overall measure of clinical severity and functioning.

Donabedian’s model conceptualizes health status as best represented by a two-dimensional matrix, in which levels of function or performance are scaled along the vertical dimension and functional domains (physical, psychological) are categorized along the horizontal dimension. For example, cardiac health status might include domains such as blood pressure, lipid levels, smoking history, and other measurable factors. Overall health status would be represented by an average across these functional domains.

To fully implement the Donabedian model in inpatient psychiatry, we need to establish a conceptual framework for “inpatient mental health status” (IMHS) and devise a method to assess it. Which aspects of mental health are relevant, and how should they be evaluated? Psychiatric diagnoses are not particularly helpful for assessing outcomes, since most patients are discharged with the same diagnosis they had at admission. More important is the severity of mental health symptoms that led to admission. The DSM-5^[7] identified mental health symptoms relevant across psychiatric diagnoses. Known as cross-cutting symptoms, they are organized into 13 domains for adults, including depression, anger, mania, anxiety, somatic symptoms, suicidal thoughts, psychosis, sleep issues, memory, repetitive thoughts and behaviors, dissociation, personality functioning, and substance use. Only some of these (e.g., depression, anger, mania, suicidal thoughts) would be considered putative causes of hospital admission. IMHS could be calculated using standardized measures of these symptom domains, with IMHS defined as the mean score across these domains.

In 2005, the Ontario government mandated the use of the Resident Assessment Instrument-Mental Health^[8] across all publicly funded inpatient psychiatric services in Ontario, including acute, forensic, long-term, and geriatric psychiatry. Since then, the RAI-MH has been used in publicly funded inpatient psychiatric care in Ontario, administered at admission and discharge, and every three months for long-term patients, or whenever a significant change in clinical status occurs. By 2020, over 1.4 million assessments had been completed for more than 320,000 unique individuals across Canada.^[9] Previous researchers have successfully used the RAI-MH scales to evaluate outcomes. ^[10,11]

An evaluation of the RAI-MH’s performance was conducted at a large psychiatric hospital in Ontario, with findings reported in a 62-page internal report,^[12] and later in the Canadian Journal of Psychiatry.^[13] The authors provide extensive empirical data and observations on the instrument, along with helpful guidance on its implementation and clinical application. However, they were skeptical of the RAI-MH outcome scales. They described the scale distributions as highly skewed, with very low baseline scores and minimal changes from admission to discharge. They doubted that the scales would be responsive to treatment effects, particularly for patients in acute care and addiction programs. Additionally, they reported low correlations between scale scores and the Global Assessment of Functioning (GAF),^[14,15] suggesting poor convergent validity. Finally, they expressed disappointment, speculating that these scales would not provide a common measure of clinical severity and functioning across a heterogeneous clinical sample.

In a previous study,^[16] we used eight outcome scales from the RAI-MH assessment in a Principal Components Analysis (PCA) to derive scores across four outcome domains: psychosis, depression, impairment (including memory), and aggression (anger). These domain scores were combined to form the Composite Index of Inpatient Mental Health Status (CIIMHS). This measure was applied to the entire population (N = 719) of a large specialized mental health hospital in Ontario at two points: the beginning (T1) and end (T2) of a three-month hospital stay, or at admission and discharge for shorter stays. We evaluated content validity by comparing the scale items with a list of symptoms, complications, or functions suspected of contributing to hospital admission. To assess concurrent validity, we compared mean scores across hospital units with varying lengths of stay and levels of clinical complexity (acute, chronic, and complex chronic). Hospital discharge data and receiver operating characteristic (ROC) analysis were used to assess the CIIMHS’s predictive validity. For construct validity, we examined changes in each measure from before to after treatment among short-term patients. Our findings provided strong evidence of validity across all four domains, leading us to conclude that the CIIMHS is a reliable and valid measure of inpatient mental health status.

The present study continues our validation of the CIIMHS, with an additional focus on its domain scores. First, we examined the relationship between the GAF and the CIIMHS and its domain scores as part of a test of convergent validity. Second, we examined changes from pre-treatment (T1) to post-treatment (T2) in each of the eight RAI-MH scale scores used in our previous paper. Third, we examined T1 vs. T2 (T1vT2) comparisons in the CIIMHS and its domain scores (psychosis, depression, impairment, and aggression), particularly in acute care and inpatient addiction programs (construct validity). Finally, we propose including the CIIMHS T1vT2 comparison in a hospital’s BSC.

Sample 1 included all patients (N = 719) receiving inpatient care at the participating hospital during one fiscal year. Scale scores were calculated at admission and discharge, or at the start and end of each 3-month quarter for long-stay patients. Subsample 1a (n = 245) participated in the hospital’s acute care inpatient program. Subsample 1b (n = 58) participated in the hospital’s inpatient addiction program. Data from Sample 1 were used to estimate treatment effects (scale score responsivity and construct validity). Sample 2 included all patients (N = 934) who received inpatient hospital care during the second fiscal year. These data were used to calculate correlations between GAF and the CIIMHS and domain scores (convergent validity), separately by fiscal quarter, in a 4x-replication, non-independent groups design. Subsamples 2a-d (ns = 393, 385, 357, and 344) were assessed during the second fiscal year in quarters 1-4, respectively.

The patients’ average age was 43 years. About 70% were male (73%, 68%). Most were never married (69%, 66%), with few married (14%, 17%) or having a live-in partner at the time of hospital admission. Additionally, 17% of both groups were widowed, separated, or divorced. Nearly all spoke English (99%). Although one-third of both groups graduated from high school and 23% had some post-secondary education, only a small percentage (8% and 14%) were employed, and 13% reported no income. The rest received a pension, social assistance, or disability benefits.

Waypoint Centre for Mental Health Care is a large psychiatric hospital (301 beds at the time) and a forensic mental health research facility in Penetanguishene, Ontario, Canada. It is one of four specialized mental health care centers in Ontario. The facility provides extensive acute and long-term inpatient and outpatient psychiatric services to the surrounding community, including Ontario’s only high-security forensic mental health program.

The current study used eight outcome scales from the RAI-MH: the Aggressive Behavior Scale (ABS), the Activities of Daily Living (ADL) Hierarchy Scale, the Cognitive Performance Scale (CPS), the Depression Severity Index (DSI), the Instrumental Activities of Daily Living (IADL), the Positive Symptoms Scale (PSS), the Social Withdrawal Scale (SWS), and the Violence Sum (VS). As in our previous study,^[16] we selected these eight scales from the 15 available in the RAI-MH based on two criteria. First, the scale indicated a possible cause of inpatient hospital admission (symptoms, complications, and functions). Second, the scale did not include static (historical) items, which could have reduced the scale’s ability to reflect change from T1 to T2. A Principal Components Analysis identified four factors (domains): Depression, which combined the DSI and SWS; Impairment, which combined the CPS, ADL, and IADL; and Aggression, which combined the ABS and VS scales. The PSS, which was unrelated to any other scale at both T1 and T2, was standardized and represented the Psychosis domain. The three multi-scale domains were created by averaging the standardized scores of each scale within the respective factor. A composite measure was then created by summing the scores for the four domains. Please see our previous study for more details about these calculations.

The GAF is widely regarded as the most commonly used measure of impairment among patients with psychiatric disorders. It is a scale from 1 (lowest level of functioning) to 100 (highest level of functioning). The GAF was introduced in the DSM-III ^[17] as the fifth dimension of its multi-axial diagnostic system but was not included in the DSM-V.^[7]

After completing the ethics review (Waypoint Centre for Mental Health Care Certificate #CRRA#12.03.01), Waypoint Decision Support provided anonymized data files containing item scores for the eight scales across all assessments for the two fiscal years. In Sample 2, the data were divided into four 3-month fiscal quarters. Separate analyses were conducted for each year (Sample 1; Sample 2).

The Sample 2 data file included GAF scores recorded at each assessment. At the end of each quarter, we identified all patients registered at the hospital and all patients discharged during that quarter. Seven hundred nineteen (Sample 1) and 934 (Sample 2) unique patients received inpatient care at Waypoint during the specified fiscal years. We then selected scale, domain, and CIIMHS scores for each patient at pre-treatment (T1) and post-treatment (T2) for each fiscal quarter, following the selection process described in detail in our previous study.

According to hospital protocol and practice, the outcome scales and the GAF were scored by different staff groups. RAI-MH outcome scales were scored by allied mental health professionals, mainly nurses, while the GAF was scored by medical staff, predominantly psychiatrists. Training and experience with the assessment instruments ensured reliable scoring. For the RAI-MH, studies of its psychometric properties have shown good reliability and validity.^[18,19] Multidisciplinary allied health professionals were trained in its use.

The Hospital’s Decision Support conducted studies of inter-rater agreement in the scoring of the RAI-MH outcome scales and found agreement among hospital raters to be “good-to-excellent.” For the GAF, instructions for use and scoring, including described anchor points, are presented in the DSM-III, and practicing psychiatrists have had both training and extensive experience in its use.

To assess convergent validity, we examined correlations between the GAF and measures derived from RAI-MH scale scores. Given skewness in these distributions, we calculated both Pearson’s R and Spearman’s Rho.

To evaluate scale-score responsivity, we performed paired t-tests comparing T1 and T2 for the eight RAI-MH scale scores in Sample 1. Additionally, using the CIIMHS and domain scores, we conducted one-way within-subjects ANOVAs comparing T1 and T2 across the entire sample (Sample 1) and separately for the Acute Care program (Sample 1a) and the Addictions program (Sample 1b).

The single-factor within-subjects ANOVA provides three estimates of variance (Sums of Squares): (1) the difference between the means at T1 and T2 (Effect); (2) differences among patients’ marginal scores; and (3) error (see Equation 1).

The variance estimate for error reflects the consistency of change from T1 to T2 across patients. The more similar the change is from T1 to T2 across patients, the smaller the error. We compared programs’ T1vT2 effect sizes (ESs). The ES is calculated as Equation 2:

Therefore, two factors determine the ES. The larger the difference between the means at T1 and T2 and the more consistent the T1-to-T2 changes across patients, the larger the ES. This makes Partial Eta Squared (ηρ2) an ideal statistical model for assessing quality of care as defined by the Donabedian model.