Kevin Boudreau

Seeing this study being widely re-shared with the take away that now AI >= human in peer review, in every dimension. People need to tone it down a little... This is primarily GPT evaluating Claude's comments. To me, the more useful signal is the 3 past editors of unnamed journals rating human vs Claude reviews, but this is only related to just 10 papers, it's not clear the editors have specific expertise in the paper topics or methods, and it reads as though the editors looked at the reviews but not the actual papers. I know we will need AI in the loop in some fashion to manage the paper deluge. I also realize that AI is already widely used in peer review and it's better to step in to manage / govern that process than let people do whatever they want. In my experience, AI certainly surfaces useful / good comments, particularly on methods, but many of the comments can be pedantic. Many models also still have terrible judgment about the other aspects that determine a study's value, e.g., whether the question is important. We're not yet at the point of ditching the human. Note: I'm pushing back here on the interpretations people are sharing when circulating this work, not the discussion in the paper itself.