Dynabench retweeted
Progress in AI continues to outpace benchmarks.
Check out this new plot, inspired by @DynabenchAI, that shows just how quickly it's happening.
Read more about it here: contextual.ai/plotting-progr…
Dynabench retweeted
Today, Meta researchers together with @MLCommons working group, are launching DataPerf, the first platform for building data & data-centric AI algorithm leaderboards.
We're excited for how DataPerf will help to push the data-centric AI field forward ⬇️
Dynabench retweeted
Announcing the BabyLM 👶 Challenge,
the shared task at @conll_conf and CMCL'23!
We’re calling on researchers to pre-train language models on (relatively) small datasets inspired by the input given to children learning language.
babylm.github.io
arxiv.org/abs/2301.11796
Dynabench retweeted
Great talk today by @max_nlp from @ucl_nlp @CohereAI on Dynamic Adversarial Data Collection at the @kclinformatics Distributed AI group!
Dynabench retweeted
Ever wanted to know more about generalisation in NLP but overwhelmed with the number of papers on ArXiv? Fear not! We read 400+ papers, 600+ experiments, and designed a taxonomy 📝 to categorise the research for you! (1/n) 🧵
arxiv.org/abs/2210.03050
Dynabench retweeted
How can we improve benchmarking? The @DynabenchAI experiment aims to make faster progress with dynamic data collection, and today, we are pleased to introduce our next stage: @MetaAI has funded 5 exciting research proposals on the theme of "Rethinking Benchmarking”! Congrats to:
Dynabench retweeted
"DataPerf: Benchmarks for Data-Centric AI Development"
What if instead of holding the data constant and benchmarking different models, we held the model constant and benchmarked different data pipelines? [1/7]
Dynabench retweeted
🧵[1/n] 😍I am very happy to present the first piece of my research on dynamic adversarial data collection (DADC). DADC leads to more trustworthy ML models. Learn how to leverage community adversarial data to make your model more robust💪🏾 huggingface.co/blog/mnist-ad…
Dynabench retweeted
Talking of the #DADC Community 🤝....Our social is tonight from 6:30-9:30pm @ a rooftop located on 1823 Minor Ave, Seattle (5 min walk from conference venue). There will be 🍕 and 🍹🧃 Spread the word 🗣️ + pls fill in the poll so we can get a sense of numbers 📊
82%I'm attending social
18%I'm NOT attending social
44 votes • Final results Dynabench retweeted
Our audience interaction @naaclmeeting transcends online-offline worlds 🤯 Who needs a metaverse when you have @boydgraber ? 😅 #NAACL #DADC22
Dynabench retweeted
We have an incredible panel lined up for you tomorrow on the Future of Data Collection! Moderated by @adinamwilliams, panelists include @annargrs, @boydgraber, @sleepinyourhat, @tongshuangwu, @laroyo, @douwekiela & @swabhz 🤩 Post any questions you have for them below! 🚀
Dynabench retweeted
Grateful for the support from @MLCommons. Join us on the 14th July at #NAACL2022 to learn more about dynamic adversarial data collection, @DynabenchAI and much much more! 🚀
DADC improves AI accuracy through higher quality and more diverse data collected with human and AI collaboration. We believe this will help the community build robust ML and why are sponsoring @DADCWorkshop @DynabenchAI at #NAACL2022
mlcommons.org/en/news/dynabe…
Dynabench retweeted
Link to paper - "Dynabench: Rethinking Benchmarking in NLP"
arxiv.org/abs/2104.14337
Also see - "Dynaboard: An evaluation-as-a-service platform for holistic next-generation Benchmarking"
proceedings.neurips.cc/paper…
Is that what we need? 🤔
Dynabench retweeted
Many of ML's major benchmarks have already become obsolete. It's getting pretty urgent to re-think ML evaluation.
Dynabench retweeted
Data collection is slow and expensive, so we give annotators a little help 🤝. Introducing Generative Annotation Assistants (GAAs) to make data collection more efficient and effective 🚀. Work will be presented at #NAACL2022 in Seattle!
Paper: arxiv.org/abs/2112.09062
[1/n]
Dynabench retweeted
We’re happy to announce our new preprint on perturbation augmentation for fairer NLP! We trained a seq2seq control-gen model to “perturb” demographic references. We use it to pretrain & finetune LMs that are fairer, without sacrificing accuracy. We measure fairness with it too 🧵

ALT 5 contributions of our research paper on perturbation augmentation illustrated with flowcharts isolating different parts of the modeling pipeline. The first is a dataset called PANDA, for Perturbation Augmentation NLP Dataset. The second is a conditional sequence to sequence model for training an automatic perturber. The third is a model called FairBERTa trained on data that had been augmented with the perturber. The fourth is a method of finetuning models on data that had been augmented with the perturber, called “fairtuning”. The final is a metric called the fairscore which measures fairness by comparing task accuracy on the original and perturber augmented evaluation datasets. There are figures describing each stage at the right.
Dynabench retweeted
🚨 New paper and datasets! 🚨
After sitting on my hands for many months 😬 I'm delighted that our #Hatemoji paper is going to @naaclmeeting! 😍🤩😎🆒
In a nutshell 🥜it uses human-and-model-in-the-loop learning 🤖🤝🙆 to tackle emoji-based hate
A 🧵 on all our new resources 1/
Dynabench retweeted
Track 1 of our Shared Task kicks off on Mon 2nd May. Do you have a knack for coming up with challenging QA examples? Prove it while competing against the AI and your peers! We will also have prizes thanks to @commons_ml. Sign up at docs.google.com/forms/d/e/1F… 🚀
#NLProc #NAACL2022
Dynabench retweeted
We are delighted to announce that the awesome folks at @commons_ml are as excited by data-centric approaches as we are, and have agreed to sponsor the DADC workshop at @naaclmeeting 🥳🥳🥳 1/


























