On a tangential note, this is why traditional pre-publication peer review isn’t very effective, and is in dire need of replacement. Meta-analytic estimates put the inter-reviewer reliability across fields at around .2 to .3, and it’s rare to have more than two or three reviewers on a paper. No psychometrician would recommend evaluating people’s performance in high-stakes situations with just two items that have a ~.3 correlation, yet that’s how we evaluate nearly all of the scientific literature!

The rest of the article is probably only interesting to people doing human-subjects psych research, but I thought this one bit was particularly revealing: I knew about the crisis in peer review, but not how bad the inter-reviewer agreement was!