New preprint!! 🚀
~27× KV-cache compression with LowRAM + quantisation + eviction, retaining 94.5% of the original LongBench score on long contexts.
Why spend memory on what you can predict? 🧵
🤖 Made with AI
LowRAM learns the key→value mapping at test time and stores corrections for the hardest cases. At 262K tokens, a separate 4× compression setup reaches up to 5× higher peak aggregate decode throughput.
More long-context requests per GPU is the real opportunity.
1
2
73
Everything is available! 🚀
📄 Paper (HF upvotes appreciated!): huggingface.co/papers/2611.0…
💻 Code: github.com/lowram-model/lowr…
🌐 Demo: lowram-model.github.io/
Huge thanks to Teresa Delgado de las Heras, Hugo Hazard, @MrTin4105 and @seawan!
Last week at @assc2026 we presented the Global Key-Value Workspace, an explicit global workspace built into pretrained LLMs. This week @AnthropicAI found evidence of one emerging on its own inside Claude, read out through a technique they call the J-lens.
The two results are complementary. The emergent workspace is approximate, its properties fixed by whatever training produced. When Anthropic ablates it, multi-step reasoning collapses while fluent behaviour survives. GKVW makes the mechanism explicit and controllable. A capacity-limited spotlight selects salient components of a pretrained model's KV cache, writes them sparsely to a shared workspace and broadcasts the result back to the whole model, iterated across layers. Multi-step maths and logic improve across four backbones.
Broadcast also shifts components toward synergy, information carried by the whole beyond the sum of its parts. Synergy is a tractable signature of the integration that Integrated Information Theory, the other leading account of consciousness, places at the centre of the story, so a workspace mechanism producing it makes the relationship between the two theories empirically testable. We reported this synergistic structure in LLMs in January, in work led by Pedro Urbina-Rodriguez and colleagues that the new Anthropic paper cites: arxiv.org/abs/2601.06851
The questions the Anthropic paper ends on, what gates entry into the workspace, whether ignition fires all-or-none, where the capacity limit sits, are exactly the dials an explicit workspace gives you. We have the testbed.
Preprint and code soon, subscribe and find more here: gw-assc-2026-poster.pages.de…
With @fredericowieser, @MrTin4105, Adnan Oomerjee and @seawan.
Presenting today at @assc2026 #ASSC29: The Global Key-Value Workspace. We build an explicit global workspace inside pretrained LLMs, show it improves reasoning, and use it as a testbed for how GWT and IIT relate. Come say hi, or subscribe for the preprint: gw-assc-2026-poster.pages.de…
Huge credits and thanks to my awesome co-authors @fredericowieser, @MrTin4105, Adnan Oomerjee and @seawan.
#ASSC2026 @theASSC
Zafeirios Fountas retweeted
🚨 New paper: AI evaluation is structurally unsuitable for continual learning (CL). To address this, evaluation should be centred on the "behavioural trajectories" that CL systems develop, with the goals of characterising possible behaviours and forecasting their evolution. 🧵
Zafeirios Fountas retweeted
Hybrid Attention Is Broken — Here’s How to Fix It 🧐
We all don't like the quadratic complexity of transformers. People have been trying to fix it, but their fixes are broken 😒
Our new paper finds that many “linearised” Transformers completely ignore their linear attention path — relying almost entirely on the sliding-window softmax (SWA).
👉 We call this component collapse — and it means most reported “hybrid” gains are misleading.
We dig deep and:
1️⃣ Expose how & why this happens
2️⃣ Show diagnostic tools to quantify each component’s real contribution
3️⃣ Propose 3 fixes that actually make hybrids hybrid again:
Inference-time hybridization
HedgeCATs (attention-weight transfer + targeted LoRA)
SSD (Scheduled Sliding-window Dropout)
✅ Our fixes recover >95% of base performance
✅ Ensure genuine linear attention use
✅ Restore attributional validity in hybrid conversions
Full paper: arxiv.org/abs/2510.05901
#AI #machinelearning #artificialintelligence #Technology #Innovation
🚀 Come find us tomorrow at #ICLR2025 in Singapore! We’ll be presenting our work:
📄 Paper: "Human-Inspired Episodic Memory for Infinite-Context LLMs"
🧠 The first approach to handle 10M+ tokens, outperforming full-context models — with zero fine-tuning!
🗓️ When: Saturday, April 26 · 15:00 – 17:30 (local time)
📍 Where: Hall 3 · Poster #211
🔗 Full paper & code: em-llm.github.io/
I’ll be there with our amazing co-author Martin Benfeghoul. Come say hi at the poster — or reach out during the conference if you’d like to chat!
#LLMs #LongContext #AI #MachineLearning #Memory #BrainInspiredAI
Zafeirios Fountas retweeted
I am glad to present the new fine-tuning algorithm that we call sparse preference optimization!
Not all tokens are born equally. Some are more important than others. Our algorithm allows for that, but more importantly, it allows for sparsity!
And our method does awesomely well!
Check it out: arxiv.org/pdf/2410.05102
#AI #machine_learning
Zafeirios Fountas retweeted
🚨🚨 New video out with @zfountas teaching us all about human-memory for #LLMs! It is amazing we discuss EM-LLM and we then talk about human experiments that align with the results of the paper! 🥳🥳🥳
Check the video: youtu.be/gWoh_5fsZpA
Check the blog: medium.com/@haitham.bouammar…
Paper: arxiv.org/abs/2407.09450
#GPT4O @ChatGPTapp #llama31 #Llama3 #Llama
🚀 Excited to share our new pre-print: "Human-like Episodic Memory for Infinite Context LLMs"! We introduce EM-LLM, a novel approach integrating cognitive science insights into LLMs for vastly extended context processing: arxiv.org/pdf/2407.09450
What we did:
· 📊 We treat LLMs' K-V cache as analogous to personal experiences and segmented it into events of episodic memory based on Bayesian surprise (or prediction error).
· 🔍 We then apply a graph-theory approach to refine these events, optimizing for relevant information during retrieval.
· 🔄 When deemed important by the LLM's self-attention, past events are recalled based on similarity to the current query, promoting temporal contiguity & asymmetry, mimicking human free recall effects.
· ✨ This allows LLMs to handle virtually infinite contexts more accurately than before, without retraining.
Our method outperforms the SOTA model InfLLM on LongBench, given an LLM and context window size, achieving a 4.3% overall improvement with a significant boost of 33% on PassageRetrieval. Notably, EM-LLM's event segmentation also strongly correlates with human-perceived events!!
We are releasing this method today with our first set of results, but more results and analysis are coming soon.
Huge thanks to all my co-authors/colleagues for this amazing collaboration: Martin A Benfeghoul, Adnan Oomerjee, @fenchri, @glampouras_NLP, @hbouammar from @Huawei Noah's Ark and Jun Wang from @UCL.
For a fuller description, check out @hbouammar's thread: nitter.cf/hbouammar/status/18127…
Stay tuned for more updates and a code release soon :) #LLMs #LongContextLLMs #EpisodicMemory #NLP #CogSci #MachineLearning #AI
Excited to present - Human-Like Memory for LLMs:
Our brain interprets its continuous experience by segmenting it into discrete events which are then stored. What is even cooler is that boundaries between those events correspond to surprise – i.e., brain is -surprised by the information it saw – We all remember surprising events don’t we?
Q: Can we enable LLMs to have a human-like memory structure?
A: You bet! This is what my fantastic team just did in our latest paper!
Q: How did we do it?
A: Here’s a short (very short summary):
1. Tokens are not handled jointly, we will chunk them into blocks via a notion of surprise that is defined with respect to the LLM - as our brains do it – the difference is that we define our notion of surprise via log likelihoods (check Section 3) in the paper.
2. While surprise splits are reasonable, we notice that the utility of elements within an event during memory recall depends on their likelihood to be used by the current query.
3. With this realization, we propose refinements of the original splits by reframing the problem form a graph theoretic perspective and proposing to maximize graph-clustering metrics (check Section 3.3)
4. This finally gives us our algorithms that: 1) Chunks tokens based on surprise in an initial step, 2) refines the boundaries by building-on ideas from graph-theory.
Q: Does it work?
A: Marvelously! Results are good, where we compare to infLLM (previous SOTA) and we show an overall relative improvement of 4.3% across various tasks, including a 33% improvement on the PassageRetrieval task.
Q: Is this Human connection just a BS selling pt?
A: Nope! Our analysis reveals strong correlations between EM-LLM’s event segmentation and human-perceived events, suggesting a bridge between this artificial system and its biological counterpart. This work not only advances LLM capabilities in processing extended contexts but also provides a computational framework for exploring human memory mechanisms, opening new avenues for interdisciplinary research in AI and cognitive science.
Check out the paper: arxiv.org/pdf/2407.09450 #AI #MachineLearning #GPT4 #LLM
Zafeirios Fountas retweeted
I will also be at the Memory in Artificial and Real Intelligence workshop (memari-workshop.github.io), presenting a poster on model-based episodic control (w/ @juliancodaforno @zfountas Qinghai Guo and @NeilBurgess10), come talk to me to find out more!
Excited to announce our #RLDM2022 workshop "Temporal Representation in RL" on 11 June @BrownUniversity ⚡️
We have an amazing lineup of speakers discussing time in biological & artificial brains incl. @RichardSSutton, Marc Howard and @criticalneuro!
See: trirl.cc/
To learn more, including how to attend the workshop remotely, check out our website and consider registering to our mailing list here: forms.gle/n7h7qCWV7DUaYaa36
1
4
⚡️Our new work with @az_prd titled "Variational predictive routing with nested subjective timescales" (#VPR) is presented at #ICLR2022 tomorrow 28 Apr at 10:30 am — 12:30pm BST.
iclr.cc/virtual/2022/poster/…
VPR is an event-based hierarchical generative model for videos… 🧵👇
Finally, VPR is based on the principles of a contemporary view of predictive coding – predictive routing – initially proposed by @AndreBastos2021 et al in 2020, and has the potential to be used as a cognitive architecture of perception, including the studies of interval timing.
1
1
I’m extremely grateful to my colleague Alex @az_prd for all the fantastic work and to everybody else involved.
For more information, check out our
Project website: vpr-model.github.io/
Video presentation by Alex: youtu.be/bjdYtTpjGQ0
And poster:
2





















