Adyasha Maharana (@adyasha10) profile picture

Research Scientist @DbrxMosaicAI. Data curation, vision+language, AI+health. Previously @unccs PhD, @Snap, @allen_ai, @AdobeResearch, @IHME_UW

Chapel Hill, NC
Joined July 2012
Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)Photo by Adyasha Maharana (@adyasha10)
Our code for generating extremely long multi-session conversations with custom personas and evaluating LLMs on the LoCoMo dataset is now live at github.com/snap-research/loc…! Come talk to us about this work at Poster session 5 (Aug 13, 16:00 - 17:30 local time) at ACL 2024
Can LLMs keep track of very long conversations? We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --> LLMs struggle to remember, reason over history, draw long-range temporal/causal connections arxiv.org/abs/2402.17753 🧵
1
11
33
4,284
🎉 Adapt-♾ has been accepted to #ICLR2024 @iclr_conf! We propose a dynamic, multi-way data selection strategy for continual VLM learning with growing instruction-tuning datasets. Stay tuned for the camera-ready version with additional results on LLMs! 🙌
🚨 Introducing Adapt-♾: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection! ▶️ New multimodal instruction tuning datasets are continuously released, often containing redundant/similar content or targeting highly varied skills (i.e., tasks). ▶️ How can we enable scalable, lifelong instruction tuning for MLLMs, where a temporal stream of multi-task, multimodal instruction-tuning datasets are continually added to the existing training pool? ▶️ We present Adapt-♾, a scalable and adaptive data selection strategy that facilitates the effective learning of MLLM on new skills while reinforcing previously learned skills over time. 📖: arxiv.org/abs/2410.10636 Thread 🧵👇
2
17
40
3,795
🎉Final work from my PhD; we explore multimodal instruction tuning in a lifelong learning setting and propose a data selection recipe for effective (re)learning of past and new skills! On that note, I have joined the amazing team at Databricks @DbrxMosaicAI :)
🚨 Introducing Adapt-♾: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection! ▶️ New multimodal instruction tuning datasets are continuously released, often containing redundant/similar content or targeting highly varied skills (i.e., tasks). ▶️ How can we enable scalable, lifelong instruction tuning for MLLMs, where a temporal stream of multi-task, multimodal instruction-tuning datasets are continually added to the existing training pool? ▶️ We present Adapt-♾, a scalable and adaptive data selection strategy that facilitates the effective learning of MLLM on new skills while reinforcing previously learned skills over time. 📖: arxiv.org/abs/2410.10636 Thread 🧵👇
2
14
55
8,769
Interesting work on investigating differences between full-finetuning and LoRa-tuning! Answered a few questions I was thinking about lately. Most interesting, but perhaps intuitive, finding is that LoRA induces less forgetting than full-model finetuning.
People think LoRA is a magic bullet for LLMs. Is it? Does it deliver the same quality as full finetuning but on consumer GPUs? Though LoRA has the advantage of a lower memory footprint, we find that it often substantially underperforms full finetuning. However, it forgets less of the base model’s capabilities. In this work, we exhaustively explore this trade-off and provide practitioners a clear view of the difference between the methods. arxiv.org/abs/2405.09673
2
3
492
I defended my thesis on data curation and graduated from @unccs last week. Grateful to advisor @mohitban47, collaborators, lab mates and countless others for the experience! It's been a wild ride in this rapidly changing ML research landscape. Excited to figure out what's next🥳
🎉🎓 Congratulations to these awesome new+old MURGeLab graduates on their hooding ceremony --> PhDs @peterbhase @yichenjiang9 @adyasha10 @swarnanlp (+ last year's @byryuer and @xiangzhou14, who joined us for this year's commencement) & MS @abhayzala7 🥳 Was a fun celebration with families+friends & many photo sessions in perfect weather + blue carolina skies 😀
3
6
36
4,749
Can LLMs keep track of very long conversations? We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --> LLMs struggle to remember, reason over history, draw long-range temporal/causal connections arxiv.org/abs/2402.17753 🧵
3
59
4
183
30,781
There are logistical challenges with collecting very long real-world conversations. So, we create a human-machine pipeline to generate them automatically (using event graphs & persona-driven agents) and then manually edit them for long-term inconsistencies (text and image).
1
6
524
Are unified VL models consistent across predictions for different tasks on the same image? Thrilled to share our @TMLRorg paper where we find that VL models show significant cross-task inconsistency in their predictions for the same image across tasks. adymaharana.github.io/cococo… 🧵
1
37
3
75
24,386
VL models should be consistent in their predictions if users are to trust them, especially in real-world deployment. We can’t rely on a model that 'describes' the image on the right as a person skiing but answers 'snowboarding' when asked 'What is the person doing?'
1
1
2
306
📢Check out our #EMNLP2023 Findings work on causal debiasing! We learn confounders by forcing information loss on diverse predictive features from a biased model and using the same for causally debiasing multimodal models. cc @vaidehi_patil_ @mohitban47 @uncnlp
🚨Happy to share our upcoming #EMNLP2023 findings paper on Causal Debiasing! We learn confounders by imposing information loss on diverse predictive features & use them to causally debias representations in multimodal models arxiv.org/abs/2311.16941 w. @adyasha10 @mohitban47 🧵
5
14
2,743
I was lucky to intern with @anikembhavi and the PRIOR team @ AI2 last year and had the best time. Awesome brainstorming sessions and the coolest people, can't recommend enough!
3
598
How to select important+diverse training data under a fixed data budget? 📢"D2 Pruning" --> represent datasets as sparse undirected graph & perform forward+reverse message passing to select both difficult & diverse samples. arxiv.org/abs/2310.07931 @prateeky2806 @mohitban47 🧵
2
36
3
124
33,070
Higher-quality & less redundant data can lead to lower test errors in models on a fixed data budget. But, optimizing for data diversity leads to selection of easier samples, whereas, selection by difficulty omits easy samples necessary for better training of models (see figure)
1
2
502
Our @huggingface @Gradio Spaces Demo of the StoryDALL-E model is now live at huggingface.co/spaces/ECCV20…! Try it out and find us online at #ECCV2022 to chat about it. arxiv.org/abs/2209.06192 @hannandarryl @mohitban47 @uncnlp (And thanks to @_akhaliq for helping set up the demo!)
Presenting our #ECCV2022 work “StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation” on adapting pretrained txt2img transformers for generating visual stories! arxiv.org/abs/2209.06192 github.com/adymaharana/story… @hannandarryl @mohitban47 @uncnlp 🧵
2
13
52
Presenting our #ECCV2022 work “StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation” on adapting pretrained txt2img transformers for generating visual stories! arxiv.org/abs/2209.06192 github.com/adymaharana/story… @hannandarryl @mohitban47 @uncnlp 🧵
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation abs: arxiv.org/abs/2209.06192 StoryDALL-E, based on a retrofitting approach for adapting pretrained transformer-based t2i synthesis models, outperforms GANs on the story continuation datasets
2
29
3
95
Story Visualization requires an agent to produce a sequence of images given a seq. of captions (narrative), but it fails to generalize to unseen plots/characters. Hence, we propose the task of story continuation, where story is conditioned on initial scene for narrative context.
1
Presenting “Multimodal Intent Discovery from Livestream Videos”! Paper: aclanthology.org/2022.findin… Code: github.com/adymaharana/Video… Talk: youtube.com/watch?v=6XrP24rt… Find me at #NAACL2022 poster session 10F today cc Quan @FranckDernoncou @david_s_yoon Trung Walter @mohitban47 🧵👇
1
9
20
Identifying procedural intent within instructional videos (how-to, tutorials) is a challenging task that remains underexplored but essential to video summarization/search/reco. We introduce the task of instructional intent extraction+identification from software livestreams.
1
Presenting our work “On Curriculum Learning for Commonsense Reasoning” at #NAACL2022 poster session PS2-5 (in-person) today & PS1-9 (virtual) tomorrow! Paper: aclanthology.org/2022.naacl-… Code: github.com/adymaharana/curri… Underline: underline.io/events/325/post… cc @mohitban47 (@uncnlp) 🧵
1
19
42
We use paced curriculum learning to sample training mini-batches with increasing levels of difficulty from the ranked dataset. We also examine the effect of adaptive curriculum where the data ranking is dynamically updated during training based on current state of learner model.
1
Check out this fantastic new tool for semantic code search by @samipddd (undergrad @uncnlp). Fast code search *within IDE* with easy copy-paste UI!
Excited to release Scotch, a semantic code search tool! Scotch is fast and gives back exact code snippets right in your IDE, so no more wandering around the web to find relevant code. Website: sdpmas.github.io/Scotch/ (cc @uncnlp) Get it for VS code: marketplace.visualstudio.com…
7
The most delightful thing I have seen in a while! 😁
This is happening in the next hour 1:45–3:15 EST in #EMNLP2021 in Oral Session 6D: Speech, Vision, Robotics, Multimodal Ground; video and session/zoom links below -- come join us to chat about using structured inputs for visual storytelling! underline.io/events/192/sess…
Presenting our #EMNLP2021 work: “VLC-StoryGAN: Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization” on using structured inputs for story visualization! Paper: arxiv.org/abs/2110.10834 Code: github.com/adymaharana/VLCSt… cc @mohitban47 (@uncnlp) 1/n
2
6
Presenting our #EMNLP2021 work: “VLC-StoryGAN: Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization” on using structured inputs for story visualization! Paper: arxiv.org/abs/2110.10834 Code: github.com/adymaharana/VLCSt… cc @mohitban47 (@uncnlp) 1/n
2
15
2
60
Linguistic, visuospatial, & commonsense structure info is important for story visualization since its inputs have an explicit narrative structure that needs to be translated into a visual story, & the captions lack sufficient spatial + world knowledge details about the scene. 2/n
1
1
Here's my Dad making bridges that are expected to last a century while I am writing code that will get replaced in the next month 🙃🙃 #ml #phdlife
Er. Ramakrishna Moharana, Chief Engineer, World Bank Project, inspected the construction of a high-level bridge over river Sona at Joda with Er. Achyutananda Das, Superintending Engineer (R&B) Circle. Strategies to resolve bottlenecks related to the approach road were discussed.
5
👇👇 This is happening tomorrow Tuesday morning 09:00–10:20 PDT in #NAACL2021 session 7C: Language Grounding; video and session/zoom links below -- come join us to chat about visual storytelling! 🙂 underline.io/events/122/sess… underline.io/events/122/sess…
Our #NAACL2021 work “Improving Generation and Evaluation of Visual Stories via Semantic Consistency” on story visualization w/ dual learning+copytransform+memory (DuCo-StoryGAN)! pdf: arxiv.org/abs/2105.10026 code: github.com/adymaharana/Story… cc @hannandarryl @mohitban47 @uncnlp 1/n
4
8