Testing for Moderators of Inter-Rater Reliability in Complex Settings As It Applies to Selection Instruments

Abstract: Inter-rater reliability (IRR), commonly assessed by intra-class correlation coefficient, is an important statistic for describing the extent to which there is consistency amongst two or more raters in assigned measures. In organizational research, the data structure is often hierarchical and designs deviate substantially from the research ideal of a fully crossed design. Previous research has also shown some evidence of existence of moderators of IRR in selection instruments. However, estimation of IRR in more complex multilevel settings accounting for possible moderators of IRR has not been fully addressed. In this work, we use mixed-effect models to estimate IRR for rubrics used to rate teacher applicants to a school district. In this complex multilevel context, we estimate within school and across school IRR and we test hypotheses about possible moderators of IRR, such as applicant type (internal or external). We also enumerate the direct effect of IRR on predictive power of the selection instrument and its components, and we offer practical applications and policy implications of the methods we employ.