
CS PhD at CIIR @manningcics, founder of @NagetInc. Researcher in NLP & Information Retrieval. Search, nuggets, search.
United States
Joined March 2020
- Tweets227
- Following1.1K
- Followers402
- Likes974
Chris Samarinas retweeted
🤖➡️📉 Post-training made LLMs better at chat and reasoning—but worse at distributional alignment, diversity, and sometimes even steering(!)
We measure this with our new resource (Spectrum Suite) and introduce Spectrum Tuning (method) to bring them back into our models! 🌈
1/🧵
Chris Samarinas retweeted
Excited to release new repo: nanochat!
(it's among the most unhinged I've written).
Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.
It weighs ~8,000 lines of imo quite clean code to:
- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.
Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.
My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.
Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.
Chris Samarinas retweeted
I’ll present our full paper "Bridging the Gap: From Ad-hoc to Proactive Search in Conversations" tomorrow (16 July) at @SIGIRConf #SIGIR2025, in the Conversational IR and Intelligent Agents session, MANTEGNA Platea, Floor 1, 10:30–12:30.
Paper: dl.acm.org/doi/10.1145/37263… – at Padua, Veneto
Chris Samarinas retweeted
Come to SIGIR Session M3.2: Conversational IR and Recommendation to hear from @CSamarinas about proactive conversational search!
#SIGIR2024
Chris Samarinas retweeted
Join me for my presentations at #SIGIR2024 M3.1 RAG session July 15 4pm
1. Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models
2. Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation
Chris Samarinas retweeted
If you are attending #SIGIR2024, come to our (@snbruch, @cosimorulli1, @rventurini_) talk in M1.3 on Seismic (efficient approx. sparse retrieval)! @snbruch drafted a nice blog post to describe the algorithm, w/ plenty of context: bruch.io/blog/publications/2…
Chris Samarinas retweeted
Today, I'll be presenting our #SIGIR2024 paper titled "Ranked List Truncation for Large Language Model-based Re-Ranking" at Session Efficiency for Search (M1.3), which starts at 10:30 am, in Room Federal A. @SIGIRConf
Paper: dl.acm.org/doi/10.1145/36267…
Code: github.com/ChuanMeng/RLT4Rer…
Chris Samarinas retweeted
I'm at #SIGIR2024 this week-- very excited to be giving a talk about our long context work at LLMs Day (Tuesday @ 12:15 in the Presidential Ballroom)!
And I would love to chat with folks interested in long context, attention mechanisms, or IR perspectives on RAG :)
Chris Samarinas retweeted
Amazing to see a conference paper search tool that goes beyond text similarity. Check it out: sigir.naget.com/
#SIGIR2024
Chris Samarinas retweeted
🚀 Scaling 10M naturally existing instructions from the Web! 🌐
Introducing 🦣MAmmoTH2 (tiger-ai-lab.github.io/MAmmo…), a family of open-source models that unlocks unprecedented performance gains in reasoning tasks by leveraging two key insights:
1️⃣ Scaling is still all you need, even in the instruction tuning phase! 📈
2️⃣ High-quality, diverse instructions naturally exist on the web and can be efficiently harvested without costly human annotation. 💸
Our 3-step pipeline mines 10M instruction-response pairs from web data:
1️⃣ Recall relevant docs from Common Crawl using fastText 🔍
2️⃣ Extract instruction-response pairs from docs using open-source LLMs 🪄
3️⃣ Refine the extracted pairs using open-source LLMs to fix formatting & add missing rationales ✨
By fine-tuning base LLMs on this dataset, MAmmoTH2 models achieve huge gains on reasoning benchmarks without seeing in-domain data, e.g.:
- MAmmoTH2-7B: Mistral-7B MATH 11% → 34% 🧮
- MAmmoTH2-8B: Llama-3-8B TheoremQA 20% → 30% 🎓
Further fine-tuning MAmmoTH2 on open-source instruction data, including OpenHermes 2.5, Code Feedback, and Math-Aug (our newly constructed), leads to MAmmoTH2-Plus.
🌟 MAmmoTH2-8×7B-Plus (Mixtral MoE base) matches 110B Qwen's reasoning performance with only 13B params while achieving impressive chat abilities, scoring 33.8 on AlpacaEval 2.0 and 32.6 on ArenaHard. 🏆
🌟MAmmoTH2-8B-Plus and Llama-3-8B-Instruct, both derived from Llama-3-base and tuned on 10M-sized datasets, provide an apples-to-apples comparison. Despite Llama-3-8B-Instruct using human-annotated data and MAmmoTH2-8B-Plus not, our model surpasses Llama-3-8B-Instruct by 6 points in reasoning tasks, showcasing the power of our annotation-free approach. 💡
Our work demonstrates a new paradigm for building powerful instruction-tuned models by efficiently harvesting large-scale, high-quality data from the web without costly human annotation or GPT-4 distillation. 💸🧠
An online demo (for reasoning and general chat; Yes, our model is not just intended for reasoning!) is available at: huggingface.co/spaces/TIGER-…
Kudos to the co-lead: @WenhuChen and the team: Tuney Zheng @GeZhang86038849!
Chris Samarinas retweeted
Here is a list of contributions from the @manningcics Center for Intelligent Information Retrieval (CIIR) to @SIGIRConf 2024!
Multiple exciting papers on RAG, Generative Retrieval, Proactive Conversations, and LLM Personalization.
Pre-prints coming up, stay tuned... #SIGIR2024
Chris Samarinas retweeted
🔥EVERYONE🔥We’re excited to announce the release of OpenAssistant.
The future of AI development depends heavily on high quality datasets and models being made publicly available, and that’s exactly what this project does.
Watch the annoucement video: youtu.be/ddG2fM9i4Kk
Chris Samarinas retweeted
1/ In 2021, we shared next-gen language + conversation capabilities powered by our Language Model for Dialogue Applications (LaMDA). Coming soon: Bard, a new experimental conversational #GoogleAI service powered by LaMDA.
blog.google/technology/ai/ba…
Chris Samarinas retweeted
Wow, holy shit this is far away from anything else I’ve seen. Insane.
Google basically solved AI music: google-research.github.io/se…
Chris Samarinas retweeted
Very excited to share my @MetaAI internship work: Unsupervised Passage Re-ranker (UPR), an approach to re-rank retrieved passages for information retrieval tasks. (1/8)
Paper: arxiv.org/abs/2204.07496
Chris Samarinas retweeted
Introducing the 540 billion parameter Pathways Language Model. Trained on two Cloud #TPU v4 pods, it achieves state-of-the-art performance on benchmarks and shows exciting capabilities like mathematical reasoning, code writing, and even explaining jokes. goo.gle/3j6eMnK
Chris Samarinas retweeted
Large-language models like #GPT3 have shown tsunami impact in #AI world. We predict profound impact of these models in countless applications to come. 100+ researchers @StanfordHAI have released a paper abt the present &future of these Foundation Models. arxiv.org/pdf/2108.07258.pdf
Chris Samarinas retweeted
The #IRAnthology helps researchers find relevant literature in the IR field. You can already check it out at ir.webis.de/
"The Information Retrieval Anthology" at #SIGIR21 by @webis_de
#ir #informationretrieval #search
Chris Samarinas retweeted
Imagine a magic window, and through that window you see another person, life-size and in three dimensions.
Project Starline is a technology project that combines advances in hardware and software to help people feel like they're together, even when they're apart. #GoogleIO
Project Starline
Chris Samarinas retweeted
Multimodal Few-Shot Learning with Frozen Language Models
arxiv.org/pdf/2106.13884.pdf
Really interesting:
1) Train an LM on text, then freeze it
2) Train an image encoder to produce *soft prompts* from images that cause the LM to output the image caption
3) Few-shot behavior emerges!






























