ICC Absolute Agreement vs Consistency: Understanding the Differences
When it comes to measuring agreement between observers or raters, the Intraclass Correlation Coefficient (ICC) is a statistical tool that is commonly used in research. ICC is a valuable measure of reliability that can be used to assess the consistency of ratings, especially when there are multiple ratings or observers. There are two types of ICC that are used, the Absolute Agreement and Consistency. While these two measures are often confused, there are distinct differences between them that should be understood.
Absolute Agreement
The ICC Absolute Agreement measures the extent to which multiple observers give the same exact score or rating to the same subject or measurement. This measure is used to determine how well observers can reach a consensus on a single measurement, regardless of whether the scores they give are high or low. For example, if five doctors were asked to evaluate the severity of a patient`s illness on a scale of 1 to 10, Absolute Agreement measures the level of agreement among them, considering only whether they gave the same number.
Consistency
On the other hand, ICC Consistency measures the extent to which multiple observers give similar scores to the same subject or measurement. This measure takes into account not only whether the scores are the same, but also how close they are to each other. Consistency is used to measure the level of agreement between raters or observers when the scores may vary in value, but still reflect a similar understanding of the subject or measurement. For example, if five teachers were asked to evaluate a student’s academic performance on a scale of 1 to 10, Consistency will measure the level of agreement between the teachers, considering how close their scores are to each other, regardless of whether they all gave the same score.
Which one should you use?
The choice of which ICC measure to use depends largely on the nature of the data and the research question being asked. If you are interested in measuring how similar the ratings of the observers are, then ICC Consistency should be used. If you are interested in measuring how well observers reached a consensus on a single measurement, then ICC Absolute Agreement should be used.
In conclusion, it is important to understand the differences between ICC Absolute Agreement and Consistency, and to select the appropriate measure depending on the research question and data. By using these measures appropriately, we can ensure that our research is reliable and valid, and that we are making informed decisions based on accurate data.