
Research Scientist @DbrxMosaicAI. Data curation, vision+language, AI+health. Previously @unccs PhD, @Snap, @allen_ai, @AdobeResearch, @IHME_UW
Chapel Hill, NC
Joined July 2012
- Tweets647
- Following714
- Followers721
- Likes1.6K
Pinned Tweet
Our code for generating extremely long multi-session conversations with custom personas and evaluating LLMs on the LoCoMo dataset is now live at github.com/snap-research/loc…!
Come talk to us about this work at Poster session 5 (Aug 13, 16:00 - 17:30 local time) at ACL 2024
Can LLMs keep track of very long conversations?
We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --> LLMs struggle to remember, reason over history, draw long-range temporal/causal connections
arxiv.org/abs/2402.17753
🧵
🎉 Adapt-♾ has been accepted to #ICLR2024 @iclr_conf! We propose a dynamic, multi-way data selection strategy for continual VLM learning with growing instruction-tuning datasets. Stay tuned for the camera-ready version with additional results on LLMs! 🙌
🚨 Introducing Adapt-♾: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection!
▶️ New multimodal instruction tuning datasets are continuously released, often containing redundant/similar content or targeting highly varied skills (i.e., tasks).
▶️ How can we enable scalable, lifelong instruction tuning for MLLMs, where a temporal stream of multi-task, multimodal instruction-tuning datasets are continually added to the existing training pool?
▶️ We present Adapt-♾, a scalable and adaptive data selection strategy that facilitates the effective learning of MLLM on new skills while reinforcing previously learned skills over time.
📖: arxiv.org/abs/2410.10636
Thread 🧵👇
Thanks to fabulous collaborators @jaeh0ng_yoon @TianlongChen4 @mohitban47 !!!
@uncnlp @unccs
📖: arxiv.org/abs/2410.10636
Code: github.com/adymaharana/adapt…
3
196
🎉Final work from my PhD; we explore multimodal instruction tuning in a lifelong learning setting and propose a data selection recipe for effective (re)learning of past and new skills!
On that note, I have joined the amazing team at Databricks @DbrxMosaicAI :)
🚨 Introducing Adapt-♾: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection!
▶️ New multimodal instruction tuning datasets are continuously released, often containing redundant/similar content or targeting highly varied skills (i.e., tasks).
▶️ How can we enable scalable, lifelong instruction tuning for MLLMs, where a temporal stream of multi-task, multimodal instruction-tuning datasets are continually added to the existing training pool?
▶️ We present Adapt-♾, a scalable and adaptive data selection strategy that facilitates the effective learning of MLLM on new skills while reinforcing previously learned skills over time.
📖: arxiv.org/abs/2410.10636
Thread 🧵👇
Interesting work on investigating differences between full-finetuning and LoRa-tuning! Answered a few questions I was thinking about lately. Most interesting, but perhaps intuitive, finding is that LoRA induces less forgetting than full-model finetuning.
People think LoRA is a magic bullet for LLMs. Is it? Does it deliver the same quality as full finetuning but on consumer GPUs?
Though LoRA has the advantage of a lower memory footprint, we find that it often substantially underperforms full finetuning. However, it forgets less of the base model’s capabilities. In this work, we exhaustively explore this trade-off and provide practitioners a clear view of the difference between the methods.
arxiv.org/abs/2405.09673
I defended my thesis on data curation and graduated from @unccs last week. Grateful to advisor
@mohitban47, collaborators, lab mates and countless others for the experience! It's been a wild ride in this rapidly changing ML research landscape. Excited to figure out what's next🥳
🎉🎓 Congratulations to these awesome new+old MURGeLab graduates on their hooding ceremony --> PhDs @peterbhase @yichenjiang9 @adyasha10 @swarnanlp (+ last year's @byryuer and @xiangzhou14, who joined us for this year's commencement) & MS @abhayzala7 🥳
Was a fun celebration with families+friends & many photo sessions in perfect weather + blue carolina skies 😀
Can LLMs keep track of very long conversations?
We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --> LLMs struggle to remember, reason over history, draw long-range temporal/causal connections
arxiv.org/abs/2402.17753
🧵
Are unified VL models consistent across predictions for different tasks on the same image?
Thrilled to share our @TMLRorg paper where we find that VL models show significant cross-task inconsistency in their predictions for the same image across tasks.
adymaharana.github.io/cococo…
🧵
📢Check out our #EMNLP2023 Findings work on causal debiasing! We learn confounders by forcing information loss on diverse predictive features from a biased model and using the same for causally debiasing multimodal models.
cc @vaidehi_patil_ @mohitban47 @uncnlp
🚨Happy to share our upcoming #EMNLP2023 findings paper on Causal Debiasing!
We learn confounders by imposing information loss on diverse predictive features & use them to causally debias representations in multimodal models
arxiv.org/abs/2311.16941
w. @adyasha10 @mohitban47
🧵
I was lucky to intern with @anikembhavi and the PRIOR team @ AI2 last year and had the best time. Awesome brainstorming sessions and the coolest people, can't recommend enough!
Apply for summer'24 internships @ PRIOR (computer vision)
@allen_ai
Join us in building large scale general purpose models for:
📷Visual Recognition
📄Multimodal Data
🤖Embodied AI
🛰️Planet monitoring
Apply by: Dec 11, '23
boards.greenhouse.io/thealle…
3
598
How to select important+diverse training data under a fixed data budget?
📢"D2 Pruning" --> represent datasets as sparse undirected graph & perform forward+reverse message passing to select both difficult & diverse samples.
arxiv.org/abs/2310.07931
@prateeky2806 @mohitban47
🧵
Our @huggingface @Gradio Spaces Demo of the StoryDALL-E model is now live at huggingface.co/spaces/ECCV20…! Try it out and find us online at #ECCV2022 to chat about it.
arxiv.org/abs/2209.06192
@hannandarryl @mohitban47 @uncnlp
(And thanks to @_akhaliq for helping set up the demo!)
Presenting our #ECCV2022 work “StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation” on adapting pretrained txt2img transformers for generating visual stories!
arxiv.org/abs/2209.06192
github.com/adymaharana/story…
@hannandarryl @mohitban47 @uncnlp
🧵
Presenting our #ECCV2022 work “StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation” on adapting pretrained txt2img transformers for generating visual stories!
arxiv.org/abs/2209.06192
github.com/adymaharana/story…
@hannandarryl @mohitban47 @uncnlp
🧵
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
abs: arxiv.org/abs/2209.06192
StoryDALL-E, based on a retrofitting approach for adapting pretrained transformer-based t2i synthesis models, outperforms GANs on the story continuation datasets
Presenting “Multimodal Intent Discovery from Livestream Videos”!
Paper: aclanthology.org/2022.findin…
Code: github.com/adymaharana/Video…
Talk: youtube.com/watch?v=6XrP24rt…
Find me at #NAACL2022 poster session 10F today
cc Quan @FranckDernoncou @david_s_yoon Trung Walter @mohitban47
🧵👇
Presenting our work “On Curriculum Learning for Commonsense Reasoning” at #NAACL2022 poster session PS2-5 (in-person) today & PS1-9 (virtual) tomorrow!
Paper: aclanthology.org/2022.naacl-…
Code: github.com/adymaharana/curri…
Underline: underline.io/events/325/post…
cc @mohitban47 (@uncnlp)
🧵
Check out this fantastic new tool for semantic code search by @samipddd (undergrad @uncnlp). Fast code search *within IDE* with easy copy-paste UI!
Excited to release Scotch, a semantic code search tool!
Scotch is fast and gives back exact code snippets right in your IDE, so no more wandering around the web to find relevant code.
Website: sdpmas.github.io/Scotch/
(cc @uncnlp)
Get it for VS code: marketplace.visualstudio.com…
7
This is happening in the next hour 1:45–3:15 EST in #EMNLP2021 in Oral Session 6D: Speech, Vision, Robotics, Multimodal Ground; video and session/zoom links below -- come join us to chat about using structured inputs for visual storytelling!
underline.io/events/192/sess…
Presenting our #EMNLP2021 work: “VLC-StoryGAN: Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization” on using structured inputs for story visualization!
Paper: arxiv.org/abs/2110.10834
Code: github.com/adymaharana/VLCSt…
cc @mohitban47 (@uncnlp) 1/n
Presenting our #EMNLP2021 work: “VLC-StoryGAN: Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization” on using structured inputs for story visualization!
Paper: arxiv.org/abs/2110.10834
Code: github.com/adymaharana/VLCSt…
cc @mohitban47 (@uncnlp) 1/n
Here's my Dad making bridges that are expected to last a century while I am writing code that will get replaced in the next month 🙃🙃 #ml #phdlife
Er. Ramakrishna Moharana, Chief Engineer, World Bank Project, inspected the construction of a high-level bridge over river Sona at Joda with Er. Achyutananda Das, Superintending Engineer (R&B) Circle. Strategies to resolve bottlenecks related to the approach road were discussed.
5
👇👇 This is happening tomorrow Tuesday morning 09:00–10:20 PDT in #NAACL2021 session 7C: Language Grounding; video and session/zoom links below -- come join us to chat about visual storytelling! 🙂
underline.io/events/122/sess…
underline.io/events/122/sess…
Our #NAACL2021 work “Improving Generation and Evaluation of Visual Stories via Semantic Consistency” on story visualization w/ dual learning+copytransform+memory (DuCo-StoryGAN)!
pdf: arxiv.org/abs/2105.10026
code: github.com/adymaharana/Story…
cc @hannandarryl @mohitban47 @uncnlp
1/n



































