ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies with Continuous Responses

2026-06-01

AI Summary

This study addresses the challenges in intraclass correlation coefficient (ICC) applications, where difficulties in selecting appropriate ICC forms and fragmented R tools often lead to analytical errors and disjointed workflows. To resolve these issues, we propose ICCDesign, a unified R package tailored for ICC reliability studies with continuous outcomes. The package introduces a novel four-step decision framework to guide ICC form selection and integrates comprehensive functionalities—including point estimation, confidence intervals, hypothesis testing, sample size calculation (based on Zou, 2012), automated reliability interpretation (following Koo & Li, 2016), and an interactive Shiny application. Built upon the ANOVA framework of McGraw and Wong (1996), ICCDesign enables an end-to-end analytical workflow, substantially enhancing both efficiency and accuracy in reliability research. The package is openly available on GitHub.

Abstract

The intraclass correlation coefficient (ICC) is among the most widely used statistics in reliability research, playing a central role in medical measurement, psychological assessment, and behavioral science. However, practical application of ICC faces two major obstacles. First, ICC can be organized into multiple forms under the McGraw and Wong (1996) framework -- including six widely reported standard forms and four additional design combinations -- and researchers must select the appropriate form based on their study design, yet existing guidelines are not always operationalized in software interfaces. Second, available R tools are highly fragmented: sample size calculation, ICC estimation with confidence intervals, and reliability evaluation are distributed across separate packages, compelling researchers to switch between tools and increasing the risk of analytical errors. This paper introduces the ICCDesign package, designed specifically to provide an integrated workflow for ICC-based reliability studies with continuous responses, assuming one continuous rating per subject-rater cell. The package integrates four core functionalities: (1) point estimation, ANOVA-based confidence intervals, and implemented hypothesis tests for supported ICC design combinations following the McGraw and Wong (1996) framework, with a built-in four-step decision framework guiding users toward an appropriate ICC form; (2) sample size planning based on Zou's (2012) closed-form formulas, supporting two planning modes and an inverse assurance calculation; (3) automated reliability evaluation based on Koo and Li (2016) criteria, with an uncertainty notification when the confidence interval spans the 0.75 good-reliability threshold; and (4) an interactive Shiny web application covering the main analysis and planning functionalities. ICCDesign is available from GitHub at https://github.com/KlariZhang/ICCDesign.

Problem

• intraclass correlation coefficient
• reliability studies
• study design
• sample size calculation
• R package

Innovation

• ICCDesign
• intraclass correlation coefficient
• reliability study
• sample size planning
• Shiny application