10.2.57 Inter-Rater Reliability Process Form

1

Policy Title: Inter-Rater Reliability Process POLICY # 10.02.57 Line of Business: Medi-Cal Department Name: Utilization Management Original Date 1/99 Effective Date 12/18 Revision Date 12/24 Governing Committee: Medical Services Committee Governing Committee Approval: Jennifer Nuovo, MD , Blue Shield Promise Chief Medical Officer
Date: 12/10/2024

Vice President (VP) Approval:
Tracy Alvarez, VP, Medical Care Solutions Date: 12/10/2024

A. PURPOSE Blue Shield of California Promise Health Plan (Blue Shield Promise) conducts Inter-Rater Reliability (IRR) evaluations at least annually, to measure the consistency in the application of guidelines, policies, or criteria performed by their licensed health care professionals and clinical staff involved in the utilization management (UM) process. The IRR process provides a mechanism for feedback on identified opportunities on an individual as well as global basis, and for recognizing quality performance.

B. DEFINITIONS N/A

C. POLICY I. Blue Shield Promise administers IRR testing annually to physician and

2

nonphysician UM reviewers using hypothetical UM test cases. The goal of this testing is to ensure staff consistently and accurately apply the evidence-based clinical review guidelines, policies, or criteria and appropriately refer cases to the Medical Directors (MD) when not meeting the criteria or when other concerns, such as quality issues, surface. II. IRR testing is administered to all licensed clinicians including nurses, physicians, social workers, therapists, or psychologists utilizing evidence- based clinical criteria such as MCG Health (MCG) guidelines, World Professional Association for Transgender Health (WPATH) guidelines, and/or medical policies, when applicable. IRR testing is not administered to non-clinicians since they do not make medical necessity decisions or apply clinical criteria.

III. The IRR evaluation process is compliant with the National Committee for Quality Assurance (NCQA) standards and is conducted annually to evaluate consistency with which physician and non-physician reviewers apply UM criteria in medical necessity decisions. All staff members applying UM criteria sets are required to successfully pass at least 2 IRR tests on an annual basis but may be required to take more than 2 IRR tests based on their functional area of clinical review. Additionally, all new staff must pass the IRR tests prior to conducting utilization review without supervision.

D. PROCEDURE: I. The IRR testing process consists of the following: a. Staff are assigned a minimum of (2) IRR test scenarios based on the types of clinical reviews they perform in UM (e.g., inpatient, outpatient, etc.) b. Blue Shield Promise Medical Care Solutions (MCS) utilizes MCG- developed IRR tests as they support compliance with federal and state parity mandates as well as NCQA requirements. c. Specific staff may take Blue Shield Promise-developed IRR tests when an MCG test is not available or applicable (e.g., Applied Behavior Analysis [ABA], dental, pharmacy, etc.). d. Minimum IRR scores of 90% must be achieved to conduct utilization reviews without supervision. e. All staff members, including new hires, are required to successfully pass any assigned IRR test scenarios prior to conducting utilization review without supervision.

3

i. If they are unable to achieve a passing score of 90% or better, they are required to immediately complete training in the area specific to the category they failed (e.g., MCG’s “On Demand Training” webinars, and/or refresher training on the clinical criteria being tested). ii. Once the specific training is completed, staff members will take an additional IRR test. iii. If they still do not achieve a passing score, further corrective action and/or an individual performance plan will be developed for the staff member. This is a collaborative effort between the manager and education team. f. Passing all IRR tests is a requirement of employment.

II. MCG IRR Development and Methodology: MCG develops IRR case studies internally by physicians and nurses using the same methods, reviews, and physician approval processes for each content solution. For example, whether testing on Ambulatory Care or Behavioral Health Care, each case study undergoes a rigorous multi-level review process that includes: a. Clinical Education review b. A second level Physician Advisor review c. IRR Product Manager review d. Copyediting, and e. Quality-control testing. Repurposed case studies are evaluated against the current version and updated as needed to ensure continued clinical validity. Case studies are released on an annual basis and the two most recent editions are supported. IRRs may be customized to align with workflows that are specific to Blue Shield Promise.

III. Blue Shield Promise IRR Development and Methodology: Blue Shield Promise MCS develops IRR case studies internally by licensed clinicians when MCG does not have applicable case studies (e.g., Dental, ABA, etc.), Clinicians developing IRR case studies use the same methods and multi-level review process including the following: a. IRR case vignettes and accompanying questions and answers follow the applicable clinical review criteria. b. IRR scenarios will be drafted by a quality review clinician with experience and training in the clinical subject of the IRR. c. All IRR scenarios will be reviewed and approved by a separate

4

licensed physician, nurse, or clinician with knowledge/expertise in the clinical scenario. d. IRR case vignettes and accompanying questions and answers will undergo clinical quality control testing by the IRR developers to ensure validity and agreement among raters. e. IRR scenarios will be reviewed and updated annually to ensure continued compliance and clinical validity.

E. MONITORING N/A

F. REPORTING I. The Inter-Rater Reliability Report will be provided annually when IRR tests are conducted or upon request. II. The results of the IRR testing will be reported to the Medical Services Committee (MSC) at the first quarterly meeting following the completion of the IRR process for the Committee’s review and comment. III. Any IRR trends or quality issues will be communicated to MCS Leadership as needed. IV. A Corrective Action Plan (CAP) will be developed when any global opportunities for improvement are identified, such as areas of inconsistency in the application of review criteria in the review process.

G. ATTACHMENTS N/A

H. REFERENCES

1. CA Health & Safety Code §1367.01 (b), §1367.045
2. MCG Interrater Reliability Case Study Overview and Methodology
3. NCQA UM Standards UM1: Program Structure
4. NCQA UM Standards UM2: Clinical Criteria for UM Decisions

5. DHCS UM Technical Assistance Guide 1.1: Utilization Management Program

   I. REVISION HISTORY

   Date Modification (Reviewed and/or revised) E-filing Number 12/24 Annual Review • Reviewed regulatory requirements • Formatting updates

5

3/24 Updated policy to reflect Medi-Cal specific requirements 10/23 • Annual review of Regulatory Requirements DHCS, DMHC, NCQA • Updated formatting 11/22 • Updated Policy Numbering to reflect Medi-Cal Only; • Annual Review of Regulatory Requirements DHCS, DMHC, • NCQA