Manan Dey retweeted
we just released: MVEB: Massive Video Embedding Benchmark 🎥
with more ai-generated videos, good video embeddings may become key, as you cant just grep through videos like for text..
Manan Dey retweeted
Thrilled our global data ecosystem audit was accepted to #ICLR2025!
Empirically, we find:
1⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024).
2⃣ YouTube is now 70%+ of speech/video data but could block third-party collection.
3⃣ <0.2% of data from Africa/South America.
1/
Manan Dey retweeted
Testing LLMs' reasoning skills is tough—human evaluations are expensive, data contamination is common, and LLM judges can be biased. We propose StructTest, the first benchmark that checks how well LLMs follow complex instructions and create structured outputs. It uses a rule-based evaluator that’s easy to adapt to new tasks. StructTest is unbiased, cheap, hard to cheat and highly scalable.
By testing structured outputs in areas like Summarization, Code, HTML, and Math—and evaluating 17 top LLMs—StructTest proves to be a challenge even for models like Deepseek-V3/R1 and GPT-4o. It’s also highly correlated with ChatBot Arena (~93%) and MMLU (>96%), making it a solid way to measure reasoning skills.
Code & Data: github.com/SparkJiao/StructT…
Paper🔗: arxiv.org/abs/2412.18011
Manan Dey retweeted
✨New Report✨ Our data ecosystem audit across text, speech, and video (✏️,📢,📽️) finds:
📈 Rising reliance on web, synthetic, and YouTube data.
🛑 80%+ datasets carry hidden restrictions.
🌍 Relative representation in languages and creators has not improved for 10+ yrs.
We're delighted to see this study covered by @Melissahei in the @techreview: bit.ly/49GEjxv
1/
Manan Dey retweeted
✨New Preprint ✨ How are shifting norms on the web impacting AI?
We find:
📉 A rapid decline in the consenting data commons (the web)
⚖️ Differing access to data by company, due to crawling restrictions (e.g.🔻26% OpenAI, 🔻13% Anthropic)
⛔️ Robots.txt preference protocols are ineffective
These precipitous changes will impact the availability and scaling laws for AI data, affecting coporate developers, but also non-profit and academic research.
🔗 dataprovenance.org/consent-i…
1/
Manan Dey retweeted
Introducing: 💫StarCoder
StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. It can be prompted to reach 40% pass@1 on HumanEval and act as a Tech Assistant.
Try it here: shorturl.at/cYZ06r
Release thread🧵
Manan Dey retweeted
Announcing a holiday gift: 🎅SantaCoder - a 1.1B multilingual LM for code that outperforms much larger open-source models on both left-to-right generation and infilling!
Demo: hf.co/spaces/bigcode/santa-d…
Paper: hf.co/datasets/bigcode/admin…
Attribution: hf.co/spaces/bigcode/santaco…
A🧵:
Manan Dey retweeted
✨Our work "How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts" got accepted at the Findings on EMNLP 2022!✨
Joint work with @manandey and our awesome mentor @koustuvsinha 🎉
Manan Dey retweeted
BLOOM is here. The largest open-access multilingual language model ever. Read more about it or get it at
bigscience.huggingface.co/bl…
hf.co/bigscience/bloom
Manan Dey retweeted
New paper alert! 🎉 Turns out you can reduce the gender biases your translation models just using relevant contexts, purely during inference! Checkout this cool work led by @evolvedeve and @manandey! arxiv.org/abs/2205.10762 [1/4]
Manan Dey retweeted
🧐🕵️I am looking for the best possible open source tool to do memory profiling!
I would like to know what part of my python code is causing these memory usage spikes that don't necessarily come from the Python interpreter.
Looking forward to reading your recommendations! 🤗
Manan Dey retweeted
We are releasing PromptSource, a toolkit for creating, sharing, and using natural language prompts.
We used it to create the largest open-source collection of English prompts: 2,000 prompts for 170 datasets!
📄 arxiv.org/abs/2202.01279
💻 github.com/bigscience-worksh…
Manan Dey retweeted
Tokenization—the least interesting #NLProc topic? Hell no! We, members of the @BigScienceW tokenization group are proud to present:
✨Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP✨
arxiv.org/abs/2112.10508
What's in it? [1/10]
Manan Dey retweeted
We’ve seen crazy interest in T0++ (pronounced "T Zero Plus Plus"), and almost 10’000 queries to the model since we announced it 3 days ago.
Probably the most hilariously decisive prediction from the model (courtesy of @_philschmid):
1/N
Manan Dey retweeted
First modeling paper out of BigScience is here!
T0 shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller!
Model: huggingface.co/bigscience/T0…
Repo: github.com/bigscience-worksh…
Paper: arxiv.org/abs/2110.08207
Manan Dey retweeted
Hi #NeurIPS2020! I and @manandey will be presenting our poster on *Evaluating Gender Bias in NLI* at the Workshop on Dataset Curation and Security today (11th Dec) at 2:30 PM EST. Drop by if you're around :)
cc: @koustuvsinha
Gather Town (Poster 19)
neurips.gather.town/app/A4ya…
Manan Dey retweeted
I'm really happy to share that our work on evaluating gender bias in NLI systems has been accepted at #NeurIPS2020 Workshop on Dataset Curation and Security. Joint work with amazing collaborators @manandey and @koustuvsinha. More details coming soon!
Manan Dey retweeted
I'll be presenting our poster on assessing viewer's mental health by analysing YouTube videos at AI for Social Good workshop at #NeurIPS2019! Drop by if you’re around! Poster sessions at 9:35-10:30 AM and 3:30-4:15 PM East MR11,12 - with @manandey
Manan Dey retweeted
Really happy that our (me and @manandey) paper has been accepted @NeurIPSConf 2019 Workshop on AI for Social Good".
We'll be discussing about the effect of YouTube videos on viewer's mental health. You can read more about our work at thechange.world/
#AI4Good #NeurIPS


























