Reliability of quantitative content analyses

Abstract

Reliable coding of stimuli is a daunting task that often yields unsatisfactory results. This paper discusses a case study in which tropes (e.g., metaphors, puns) in TV commercials were analyzed as well the extent and location of verbal and visual anchoring (i.e., explanation) of these tropes. After a first coding round, low interrater agreement scores were found. Examples are presented which demonstrate the reliability issues. The paper concludes with an overview of options to deal with reliability problems as described here. Low interrater agreement scores may reflect the richness of the data and their inherent ambiguity and need not necessarily be disapproved of.