CATEKAPPA: An R Shiny Application for Design and Analysis of Consistency Tests Based on the Kappa Statistic for Categorical Responses

2026-06-05

AI Summary

This study addresses the challenges of complex sample size calculations and difficult interpretation of Kappa coefficients in inter-rater agreement analysis, particularly for researchers lacking programming expertise. To this end, we developed an open-source, interactive web application based on R Shiny that integrates the core functionalities of the kappaSize and irr packages into a unified platform for the first time. The tool supports computation of Cohen’s, Fleiss’, and Light’s Kappa statistics and automatically interprets results according to the Landis & Koch benchmark scale. Offering both a graphical user interface and command-line access, it substantially lowers the technical barrier to conducting agreement analyses. Published on CRAN, this application provides an efficient, user-friendly, and reproducible solution for consistency assessment in categorical data.

Abstract

The kappa statistic is the most widely used measure of inter-rater agreement for categorical data. Despite its popularity, applied researchers often encounter two major hurdles: (i) determining the sample size required to achieve a desired level of agreement with given power, and (ii) computing appropriate kappa coefficients with proper interpretation. Existing R packages such as irr and kappaSize provide these functionalities but require programming skills and lack an integrated, user-friendly interface. We present CATEKAPPA, an R package that bridges this gap by combining sample size planning (via kappaSize) and agreement analysis (via irr) into a single Shiny-based web application. The package supports Cohen's kappa for two raters, Fleiss' kappa for three or more raters, and Light's kappa, and provides automatic interpretation using the Landis & Koch scale. Users can either launch an interactive graphical interface or use command-line functions for scripting. The package is freely available on CRAN.

Problem

• kappa statistic
• inter-rater agreement
• sample size determination
• categorical data
• agreement analysis

Innovation

• Shiny application
• kappa statistic
• inter-rater agreement
• sample size planning
• user-friendly interface