artifacts_project

<Unpopular opinion> Inter-reviewer anonymity is a big cause for low review quality at mega #AI conferences (e.g. #AAAI #NeurIPS #IJCAI #ICLR #ICML..). When reviewers don't have to worry about the censure of the other peer reviewers looking at the same paper, there is little downside to bad reviewing. [Note that we are of course not talking about reviewer names being visible to authors; just among the reviewers of the same paper] When I started reviewing for AI conferences, the reviewers looking at the same paper will all know each other's names. When you have to stand behind your opinions in front of your peers, there is a natural moderating force at work Over the time, however, there was a movement to anonymize the reviewers among themselves (as well as, in some conferences, at the senior PC member level). The justification given for this (at least in the AAAI/IJCAI family of conferences) was that junior reviewers will be intimidated into writing , and discussing what they really think about the paper . This justification may have had some support at one time, but I contend that it barely makes sense in the current AI Zeitgeist. To begin with, everyone knows that very few senior folks actually review for conferences anymore. Much of the reviewing is really done by junior researchers--often graduate students who themselves haven't had much experience in publishing (because, face it, we have exponentially growing number of papers). Secondly, in most conferences, inter-reviewer discussion has really become negligible (see nitter.cf/rao2z/status/170508056…). So why do we still have inter-reviewer anonymity? I contend that the knowledge that your reviews will be seen along with your name by other 3-4 reviewers of that paper may well help reign in the short/contentless/spiteful "why didn't cite my paper?"/chatGPT generated reviews. Currently, the downside for putting up such reviews is quite low--as unless the review is truly and horrendously egregious, it is not likely to get flagged to a level where it will be associated with the reviewer name. Even then, there is little practical/legal censure options available to conference organizers. (I heard countless discussions about keeping track of longitudinal reviewer behavior--but face it, AI conferences are not reddit with easy to accumulate/track karma!) We all complain eternally about reviewing quality. But we can't change the number of papers being submitted, or the fact that we can't do reviewing with just people who have a significant reputational loss to fear (there just aren't enough of them). This inter-reviewer anonymity is something we can actually change.. If you agree, consider retweeting.