The Institute for Replication (I4R) works to improve the credibility of science by promoting and conducting reproductions and replications.
Joined December 2021
- Tweets2.5K
- Following0
- Followers7K
- Likes703
I4R retweeted
How robust is development economics? 66 randomly selected papers from the AER, AEJ:AE & JDE. 66 (human!) replicators. Pre-specified robustness checks – mostly conservative ones. Replication reports will now appear on a rolling basis. @I4Replication #R2EDev i4replication.org/blog/repro…
I4R retweeted
Forensic metascience is growing fast, but it's scattered. This question came out of a conversation with the founder of @MetascienceObs: what if we worked like one field, with a clear division of labor?
Numbers: the Forensic Metascience AI Toolkit from @MetascienceObs runs GRIM, SPRITE, statcheck and Table 1 checks on a paper's own figures.
Images: @MicrobiomDigest and the image sleuths.
Replication: @I4Replication reruns published analyses.
Retractions: @RetractionWatch keeps the record.
Citations: SaneCite checks what a paper cites. Across 51,626 citations to retracted papers, AI readings found 40.7% still treat the retracted work as support (pending human review).
One banner, shared lists, no duplicated work. Who's missing from this map?
Check any paper's references free: sanecite.com/checker#paper
I4R retweeted
The people most vocal about errors in econ papers (there are many!) are motivated, and trying to take down specific people (e.g. Zucman), or debunk the field as a whole.
If we aren't willing to criticize ourselves / each other, we do these people a favor by making them look like the only truth seekers.
Tons of econ papers have errors, and those errors will come to light. This moment is a test of our belief that econ is the most truth-seeking of the social sciences.
Trying to bury and obfuscate the errors will hurt us in the long run. Some papers will need retractions, all will need errata.
It's an individual nuisance—I'm busy too and don't want to be bothered to go and re-replicate and audit all my past work. But if we don't show a willingness to cleaning up our own house, we can count on our opponents to do it.
Whether we will be more honest about our errors than our opponents remains to be seen. I suspect the risks of admitting you were wrong are probably lower than you think. It will take some brave authors to lead the way and set the standard.
I4R retweeted
Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published.
Some open questions:
1. How good is today’s AI for this?
2. How much would it cost?
3. Do you get most of the value with a targeted subset of research?
4. Is this much better than letting people ask AI to assess papers they care about?
5. What to do with papers whose data isn’t available?
6. What kind of organization would be best to lead the project?
7. What do you do with the outputs of the project?
My (preliminary!) thoughts on the above:
1. How good is today’s AI for this?
I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it.
2. How much would it cost?
ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide.
3. Do you get most of the value with a targeted subset of research?
Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years?
4. Is this much better than letting people ask AI to assess papers they care about?
I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say.
5. What to do with papers whose data isn’t available?
In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort?
6. What kind of organization would be best to lead the project?
A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization?
7. What do you do with the outputs of the project?
Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?
I4R retweeted
We have another @I4Replication report on #GDRI_rep. This time we replicate a paper originally published in the Economic Journal. Author responses can be found in the appendix. econstor.eu/bitstream/10419/…
I4R retweeted
This might be a first in economics: Authors officially request to be removed from a paper published in REStat. The paper itself has not been retracted - and the other authors remain on the paper. direct.mit.edu/rest/article/…
I4R retweeted
The 2027 BITSS Annual Meeting call for speakers is open until Nov 1! The BITSS AM brings together academic researchers, donors, and policymakers to discuss research transparency, rigor, methods, and ethics.
🗓️ Fri, Feb 26, 2027
📍Berkeley, CA
Apply: cega.submittable.com/submit/…
I4R retweeted
Brilliant work here, and the thread too. Something we've had to grapple with a bit in developing PLMS. The discussion in the paper on indirect PII, which I think is the practically thorniest part of deidentification, is particularly thoughtful and helpful. Read the whole thread
25/ Have a look at the paper here:
econstor.eu/bitstream/10419/…
Blog posts:
i4replication.org/blog/when-…
i4replication.org/blog/what-…
I4R retweeted
Very important ! Personally identifiable information is being uploaded to repositories. Please read and share widely. Thank you @I4Replication!
I4R retweeted
Another nail for the IV coffin. Glad to see this paper @i4replication. econstor.eu/handle/10419/343…
I4R retweeted
We have an excellent line-up for our Leibniz Open Science Day on Nov 17 in Berlin - including an exciting keynote by Felix Holzmeister. Come join us and catch up on the latest in meta-scientific research!
zbw.eu/de/ueber-uns/veransta…
I4R retweeted
EJMR is dead; here is The Economist: “Why the economics profession doesn’t trust Acemoglu.” Trivia: they cite our expert review of Acemoglu et al. (2001). economist.com/finance-and-ec…
I4R retweeted
The debate about Borjas/Breznau's paper on ideological bias in migration research is out now @I4Replication. With a comment by Katrin Auspurg & Josef Brüderl: hdl.handle.net/10419/341575 And a reply by the original authors: hdl.handle.net/10419/341576
I4R retweeted
Happy to have the CBS Replication Games (and my replication research more broadly) featured on the ODISSEI Podcast! Tune in if you're interested in hearing
Tom Emery and I nerd out over replication!
odissei.transistor.fm/episod…
I4R retweeted
The reply to our comment is now online at AEJ:Applied. The broader takeaway: should we trust an IV if removing coding errors requires a different first stage? mollylipscomb.com/_files/ugd…
New @I4Replication DP: We replicated the seminal Lipsomb et al. paper on electrification in Brazil - and found consequential coding errors. Turns out the paper is based on a fragile IV. After many years, the replication is forthcoming in AEJ:Applied. econstor.eu/bitstream/10419/…
I4R retweeted
New DP @I4Replication: An Empirical Assessment of Computational Reproducibility in Sociology. econstor.eu/bitstream/10419/…
I4R retweeted
The authors of the great Star Wars paper and I just posted a 1-page corrigendum. It does not change conclusions. Very fun working with my fellow Jedi meta-scientists. That I checked the work is a measure of the paper's importance. The Force aligned us.
I4R retweeted
This is a fantastic and rare (!) example of authors of a published paper and the author of a comment on that paper working together to fix the scientific record. Kudos to all involved. This is not easy, even with good faith all around. I hope to see the AER follow suit before too long. @davidroodman @AEAJournals @I4Replication
I4R retweeted
New @I4Replication DP: We replicated the seminal Lipsomb et al. paper on electrification in Brazil - and found consequential coding errors. Turns out the paper is based on a fragile IV. After many years, the replication is forthcoming in AEJ:Applied. econstor.eu/bitstream/10419/…
I4R retweeted
We’re approaching DP #300 at @I4Replication. Send us your latest meta-scientific paper to claim the anniversary spot. Can someone run a synthetic Econstor/OSF/arXiv exercise to estimate how many of the 299 DPs would not have appeared without I4R? We promise not to replicate you.
