Why can Agree produce values larger than one? #164
Description
I have the following dataset containing 12 subjects rated by 104 raters: ratings.csv
The raters are allowed to rate a subject using 3 categorial variables 1,2 and 3 (although 1 has never been chosen by any rater/ subject combination).
If I calculate the simple percentage agreement (using some tidyverse methods here to import and filter the data), I get a value of 2:
Agree(read_csv("ratings.csv") |> select(!study_object) |> mutate(across(everything(), as.factor)))
Rows: 12 Columns: 105
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (1): study_object
dbl (104): 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 4...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
[1] 2
attr(,"subjects")
[1] 12
attr(,"raters")
[1] 104
This seems unrealistic to me, because I think the max percent agreement should be 1 which is only the case when every rater agrees on every subject.
As this is not the case here, I would expect the value to be 0 <= x < 1.
Can you maybe help me on this one and explain the (to my eyes) strange behavior?