QI in Action: Bridging the Gap - Harnessing OCAN data for quality improvement

The Ontario Common Assessment of Need (OCAN) is a standardized assessment instrument, that the community mental health sector has used since 2009. This tool supports conversations between service providers and their clients. It helps service providers to understand the client’s current situation, needs and strengths, and to develop a service plan. OCAN assesses needs in 24 areas, called “domains." It supports a recovery approach with the inclusion of a client self-assessment component.

At a recent virtual train-the-trainer session hosted by EENet and Ontario Health, Frank Sirotich discussed the results of a recent project, which looked at how OCAN data could help identify an opportunity for quality improvement (QI). Frank is Director of Data Analytics, Research and Evaluation at the Canadian Mental Health Association (CMHA) Toronto.

This edition of QI in Action features highlights from his presentation in his own words, edited for clarity.

The context

Sometimes it feels like we put a lot of work into collecting OCAN data. It is relevant for clinical practices, but it is a lot of work for service providers to complete and can sometimes feel like it goes into a black hole.

I know a lot of organizations are looking at how they can use OCAN data for program planning and for QI. The presentation today is an example of how we can bridge that gap between using the assessment data for direct clinical care, on the one hand, and using it for QI and planning purposes, on the other.

Today, I’m going to present about a QI project that we did at CMHA Toronto and I’m going to discuss the following topics:

At CMHA Toronto, we knew from the research that unmet physical health needs were associated with a greater risk of death, lower quality of life, more emergency department and inpatient use and longer time in service. Therefore, in our previous strategic plan, we prioritized developing our primary care capacity and invested in some programming for this purpose.

The problem

OCAN data for individuals who had at least four assessments within our case management services showed that rates of unmet physical health needs over time did not change after we implemented the new programming. Therefore, we decided to make addressing the lack of change our first project with the Excellence through Quality Improvement Project (E-QIP).

Sources of information and QI approaches for diagnosing root causes

We used QI approaches to help us identify underlying causes for the lack of change. The process involved the following steps:

  1. We developed an aim statement for this QI project, which was to reduce unmet physical health needs by 20 per cent in a case management program.
  2. We used a variety of QI tools to help us diagnose the problem. These tools were a fishbone exercise to generate some hypotheses and a staff focus group to hear what staff thought might be contributing to the problem.
  3. With a combination of input from staff and the fishbone exercise, we developed questions for a survey that asked clients what they thought might be contributing to the problem.
  4. Finally, we used a Pareto chart to analyze survey results and identify and prioritize the most significant factors contributing to the problem. This helped narrow down our focus to a few key causes.

Potential root causes

The information we gathered pointed to four root causes:

  1. CMHA Toronto primary care services did not entirely meet all client preferences as they focused on health promotion and physical activity. We did not yet have funding for a full-time nurse practitioner to provide direct physical health care.
  2. Staff did not have a clear understanding of how to rate unmet physical health needs in the OCAN.
  3. Clients either did not have a primary care provider (e.g., a GP or other primary care practitioner) or had to wait a long time to see one. Even when they finally had an appointment, it was not uncommon for them to spend two or three hours in the waiting room, which dissuaded them from going.
  4. Finally, clients were not aware of the importance of getting regular treatment for chronic physical health conditions and comprehensive physical health exams.

Figure 1

Drivers needed to achieve key change ideas for intensive case management clients

Figure 1 Aim and Big Dot: Decrease unmet physical health needs for ICM clients by 20% by October 31, 2023. Primary Drivers: Increase convenience and scope of PC services; Staff better able to identify unmet need; Increase staff and client knowledge of PC services and motivation to access services Secondary Drivers:  Increase 1-to-1 PC services at CMHA; Offer PC services at more convenient location; Increase consistency using OCAN to identify unmet needs; Share information about PC services to staff/clients; Enable staff to monitor progress addressing unmet needs; Enable staff to increase motivation of clients to access PC services; Develop clear pathway of how/when to access CMHA PC services Change Concepts/Ideas: Increase step care to one day/week NP lead clinic at main office site; Create 1:1 health talk and education with NP at main office; Provide guidelines on identifying unmet need; develop screener that includes definitions of need; E-module training on what met vs. unmet need is; Build on educational/resource wall outside PC office; include PC resource section on intranet; Team visits to discuss PC referral and screening protocol; Provide regular OCAN reports tracking needs over time; Train staff in motivational interviewing; Create standardized pathway re: specific steps to refer to PC; Dedicate time each team meeting to discuss clients with unmet PC needs. Key change ideas: 

Change ideas

Once we understood the possible root causes of the problem, we developed a driver diagram (Figure 1), which is a framework outlining the project objective (e.g., to reduce unmet needs by 20 per cent). We also identified key drivers that we needed to put in place and specific change ideas that we could implement to try to achieve that objective.

Since we had no new dedicated funding or resources, we selected two change ideas that would require the fewest resources and would have the greatest impact. These were:

Table 1: Defining unmet and unknown physical health needs in the OCAN

Unmet needUnknown need
Client has a diagnosed physical health condition.Client and care team give no information about physical health. 
Client has not been following treatment recommendations. Client has had no comprehensive physical exam in the past year. 
Client has followed treatment recommendations but finds it is largely not helpful.  
Client has no diagnosis but has a physical health concern. 

Process map

We then developed a process map of standardized responses for case managers based on the client’s physical health need rating (Figure 2). Key pathways in the process map identified the following three interventions for clients with unmet or unknown physical health needs:

  1. If a client had an unmet physical health need, their case manager would try to refer them to a doctor and bring that case to a monthly team meeting to consult with a nurse practitioner.
  2. If the client had an unknown physical health need, we would facilitate a referral to their primary care practitioner.
  3. If the client declined seeing a primary care practitioner, their case manager would bring the case to the monthly team meeting for discussion with the nurse practitioner, to identify ways to motivate the client to see a primary care provider.

Figure 2

Three clinical response pathways for case managers

Start Primary Health (PH) Care Screener. Does client have PH condition/serious PH concern? (Yes/No) Is client receiving/adhering to treatment for PH condition? (Yes/No) Is treatment not helpful at all or largely unhelpful? (Yes/No) Rate OCAN PH need as unmet and bring case to team for case conference with NP. Bring case to team for case conference with NP. Determine plan/approach and when need can be rated as met; review progress with team monthly. Second row: Has client had PH assessment in past year? (Yes/No) Does client have Dx of schizophrenia and started new anti-psychotic in last 12 weeks? (Yes/No) Did client receive PH Assessment in last 12 weeks? (Yes/No) Rate OCAN PH need met and review at next OCAN assessment. Third row: Refer client to local PC provider and rate OCAN PH Unknown. Does client refuse to see PC providers for PH assessment? (Yes/No) If yes, Bring case to team for case conference with NP. Determine plan/approach and when need can be rated as met; review progress with team monthly. If no, support client to see PC provider; review PH status after sees PC provider.

The physical health needs of clients who had unmet or unknown physical health needs in a given month were re-assessed in the subsequent month. If there was no change in their need rating, their case was brought back to a team meeting for further consultation.

Measuring change with the OCAN

We also sought to find a way to measure the impact of our change ideas using the OCAN and focused on the following two approaches:

  1. measuring change in unmet and unknown need ratings each month
  2. measuring overall change in unmet and unknown need ratings over the course of the project.

We collected information about physical health needs on a month-to-month basis because we recognized that waiting a full six months for a new OCAN cycle would be too long for a QI project. Each month, case managers used the definitions in Table 1 to rate the physical health need status of any client who had an unmet or unknown need in the previous month. We then counted the total number of clients who had physical health needs that remained unmet or unknown from the previous month (negative outcome) and the total number of clients whose need status changed from unmet or unknown to met or no need (positive outcome). With this information, we calculated the unmet needs ratio, calculated by dividing the negative outcomes by the total of the negative outcomes plus the positive outcomes (Figure 3). The lower this unmet need ratio, the better the outcome.

Figure 3

Unmet need ratio formula

Unmet need ratio= Negative Outcome / Positive + Negative Outcomes 

Example of unmet need ratio for December 2019:

  • 11 negative outcomes
  • 5 positive outcomes 
  • Unmet need ratio = 11/(5+11) = 0.69

For example, in the month of December, we had five clients whose physical health needs changed from unmet or unknown to met or no need (positive outcome) and 11 clients whose physical health needs remained unmet or unknown from the previous month (negative outcome). To calculate the unmet need ratio for that month, we divided the number of clients with negative outcomes (11 clients) by the number of clients with either positive or negative outcomes (16 clients) which equals 0.69.

With this information, we created a run chart, which is a QI tool that allowed us to track the unmet need ratio’s progress over time to determine if change ideas were having any impact (Figure 4).

Figure 4

Ratio of change in unmet physical health needs

Unmet need ratio. October 2019: 1.0. November 2019: 1.0. December 2019: 0.69. January 2020: 0.65. February 2020: 0.67. March 2020: 0.73. April 2020: 0.73. May 2020: 0.73. June 2020: 0.73. July 2020: 0.65. August 2020: 0.80. September 2020: 0.75. October 2020: 0.80. Median unmet need: 0.75.

Next, we looked at total change over the course of the project (Figure 5). Here, we looked at individuals who had at least two OCAN assessments in the course of the project. This showed that unmet need had decreased by 23 per cent and unknown need had decreased by 82 per cent. This showed that the change ideas had a greater effect on getting clients to see their doctors than in addressing their unmet physical health needs.

Figure 5

Change in physical health need status from first to most recent OCAN assessment

Number of unique individuals. Unmet need. Baseline: 13 individuals. Current: 10 individuals. Unknown need. Baseline: 11 individuals. Current: 2 individuals.

Lessons learned

There were several lessons learned from this QI project:

The overall takeaway is that the OCAN, in addition to having value in clinical practice, has applications for QI purposes and for planning.

See more related to