Sojun Park

Replication has become much easier in the era of generative AI. I'm not the first person to say that. However, I've seen fewer people acknowledge a specific aspect of this lowered cost for replicating scientific work: Generative AI will very soon allow us to assess the robustness of individual scholars' full bodies of work. Soon, we will be to compute measures of which scholars do robust science, and which do not. What's wild is that we may be able to almost do that already. Let me show you what I mean. In June, I gave Claude a pretty basic prompt. It read: "I have a big task for you. I want you to start a folder. Call it Acemoglu Replications. Then, go find as many replication archives for Daron Acemoglu as you can. Keep a spreadsheet of the ones you can find and those you can't. Then, start a replication/reproduction effort on those articles. People have in the past criticized the research designs and general robustness of his individual papers. I want to know how strong his body of work is as a whole. Don't come in with any prior beliefs; be dispassionate." I let Claude run overnight while I slept. When I came back in the morning, 29 of Acemoglu's replication archives were fully loaded in my Dropbox. All the code reproducing the paper's results had run. And there was a first draft of a paper assessing the robustness of Acemoglu's full body of empirical work. I'll admit, the first draft of the paper wasn't great. But with 15 short follow up messages--which took me about an hour to write--I was able to prompt engineer a paper-length examination of Acemoglu's work. I've attached the screen shot of the abstract below. I think this reassessment of Acemoglu's work is certainly not done. I'm posting the abstract as a proof of concept, rather than a definitive answer. I'm not posting the full paper yet because I think it still needs more work. Ultimately, I paused this project for three reasons. 1.) Limited time/topical expertise: Most of Acemoglu's work is outside of my area of topical expertise. So, I have limited time to work on it. What this type of a project really needs is someone who has the time and the know-how to dig into each of the replication's individually to make sure they are doing the right things. I think the ideal approach combines the breadth that LLMs afford and the depth of attention/expertise that humans can give. 2.) Questions about the value of the "assess one scholar at a time" enterprise: I totally get that having a database of scholar-level robustness metrics would be very valuable in theory. But what I don't know is whether this approach is truly valuable. Moreover, doing so would come with distinct challenges. a.) Many journals have very restrictive space constraints. A body of work approach would, of necessity, be very long. b.) Collecting replication archives is harder for some types of scholars (those who post them all on their websites) than others (those who don't). c.) We'd have to think hard about questions like: what scholar-specific robustness metrics would be best? And: how would we deal with the fact that prolific authors' robustness metrics would be estimated much more precisely than less prolific scholars? Additionally, I'm just not sure that "taking on" one scholar at a time has enough scientific merit to pursue. If I measured how robust an individual scholars' work is, I'd ideally want to know where that metric stands vis-a-vis the rest of scholars in that field/area. To do that, we'd ideally want the population of these scholars or, at minimum, a random sample. Concretely, if Acemoglu has, say, 78% of published headline results reproducible under some standardized protocol, is that excellent, mediocre, or terrible? To answer that, you need a reference distribution. That makes a random or otherwise well-defined sample of scholars much more attractive than selecting prominent individuals one by one. (I'll acknowledge that I may just be wrong on #2. Arguing against myself, I do agree that human-driven reproduction/replication work rarely assesses full/representative slices of a field. Instead of assessing one scholar at a time, we assess one paper at a time. Field-wide detective work is becoming more common, but my sense is that it's still the exception rather than the rule.) 3.) Cost/benefit considerations and replication norms: we have very weakly formed norms around reproduction/replication generally speaking. We have basically no developed norms around replicating individual authors one at a time. What this means is that the people who would lead a scholar-by-scholar replication effort will, likely, bear a heavy cost and, potentially, reap limited benefits. On the costs side, focusing on scholars' total bodies of work risks making the replicators look petty, vindictive, and antisocial. Enough of the scientific field is hostile towards replications of individual papers. Imagine what will happen if/when a scholar submits a scholar-specific "take down" of a full body of work. My sense is that it's common enough for scholars having their work replicated to be asked to be a reviewer for those manuscripts. I've seen very hostile responses when one paper is at issue. Imagine what type of reviewer Acemoglu would be for a paper that took on his entire body of empirical work! Even if Acemoglu weren't a reviewer, prolific authors tend to have wide coauthor/friend networks. The rally-around-my-friend dynamic we often see would certainly work against this type of paper being published. Even a completely neutral analysis acquires an accusatory character simply because the sampling unit is a named person. And that creates an unfortunate problem of its own: readers may interpret the choice of scholar as evidence that the investigators expected to find something. On the benefits side, replicating individual scholars' total body of work may offer limited payoffs. What journals would accept this type of scholar-specific replication? I'm not sure the top ones would. Conclusion: Generative AI has enormous potential in assessing and, ultimately, enhancing the robustness of scientific research. Instead of asking questions like, “does this famous individual paper replicate?”, we can begin asking questions like: -“What proportion of published empirical findings in [field X] survive a common robustness protocol?” -“How much of the variation in replicability is attributable to papers, authors, journals, methods, or subfields?” -“Are scholars persistently more or less robust across their work?” -“Can we predict which findings will prove fragile?” I may just be wrong on what I think about a one-at-a-time full body examination of scientific research. If I am, please let me know! I am also happy to chat one-on-one with anyone who is curious to learn more about the early-stage Acemoglu-specific replication project.