Ala Falaki

📝 Language Models Can Control Their Own Attention What if AI could decide which parts of a long document to read instead of looking at everything for every word it writes? This paper shows how language models can take control of their own attention, and the idea is surprisingly simple. Normally, AI models look at the entire context every time they generate a new word, even when most of it is not needed. This new method help the model to first find the useful part of the text, then focuses only on that part to answer the question. The model starts in global mode, where it looks at the full context to find the useful parts. It then switches to focus mode to read only those parts and extract the needed information. Next, local mode lets the model think using what it has already found, without reading the full context again. Finally, it uses that information to give the answer. Basically, this setup means the model generates fewer words using expensive full attention (just mention which chunk it should look at), while most of the answer uses cheaper attention (just read a 2K token chunk to generate the answer). This only happens during inference, without any extra training, and focus mode directly blocks attention to the parts of the context the model does not need. The method reduces attention work by 45.8% on Gemma and 19.6% on Qwen. Accuracy drops by only 1.4 and 2.1 percentage points. Some generation steps become up to 55% faster on Gemma. The method also saves more attention work as the context gets longer.

📝 Copy-Paste Language Models LMs often need to repeat words or information that already appeared earlier in the text, but doing so is not always easy. This paper explores a simple way to help language models reuse information from their context more effectively. Traditionally, language models predict each new token using a standard output layer called the LM head. Even when a token already appears in the context, the model must predict it through this layer. Previous methods used attention mechanisms to find and reuse information from earlier text. However, the LM head can limit both what the model can predict and how efficiently it learns (look up softmax bottleneck and LM head gradient bottleneck). To address these limitations, the authors introduce Copy-Paste Language Models (CPLMs). They add a new Copy-Paste head alongside the standard LM head, allowing the model to copy tokens directly from earlier text. The LM head decides when to use copying through a special <copy> token, while a <sink> mechanism lets the model fall back to normal prediction when copying is not useful. Both heads are trained together, without needing any extra training labels or a separate loss function. CPLMs reach the same training loss up to 2× faster than standard models. Adding the Copy-Paste head to a standard sliding-window Transformer improves its average math, coding, and instruction-following scores from 18.9% to 24.0%. They shine in long-context tasks, where retrieval accuracy jumps from 14.3% to 99.4%. They also match full-attention models while using 12× less memory per token. This works because the Copy-Paste head can directly find and reuse information from earlier text, while also helping the model learn more efficiently.

It's strange that @Medium is forcing writers to buy membership if they want to get paid, while their payout wouldn't even cover tenth of the membership cost. Now, lots of writers will post their pieces for free so there will be even less incentive for readers to get the subscription, or might not even post them on Medium anymore!

📝 Zep: A Temporal Knowledge Graph Architecture for Agent Memory Zep explores how AI agents can remember information from past conversations and use it in future interactions. It focuses on the challenges of maintaining accurate and useful memory as information changes over time. The paper introduces Zep, a memory system that uses a knowledge graph to organize information from conversations. It stores the original messages, extracts important facts and relationships, and groups related information together. When new information arrives, it uses an LLM to update facts while keeping track of what was true in the past. When the agent needs to remember something, the system searches the graph and retrieves the most relevant information to help generate an answer. Graphiti is the engine behind Zep that builds and updates a knowledge graph to organize facts and relationships. By keeping track of when information was true, Zep can retrieve relevant memories and understand how things have changed over time. It achieved 71.2% accuracy, compared to 60.2% when using the full conversation history. It also reduced response time from 28.9 seconds to just 2.58 seconds. The system used only 1,600 tokens instead of 115,000 tokens. It performed especially well on user preferences, improving accuracy from 20.0% to 56.7%, and temporal reasoning, improving from 45.1% to 62.4%.

📝 Memory Mosaics at scale This paper introduces Memory Mosaics v2, a new language model architecture that uses associative memory instead of traditional Transformer attention. It explores how this approach can help AI models store and retrieve information and learn new tasks from examples. The architecture has three types of memory: short-term, long-term, and persistent memory. Short-term memory focuses on recent information, while long-term memory stores information from earlier parts of the text. Persistent memory keeps the general knowledge learned during training. The model uses attention-like scores to find and retrieve relevant information from its memories. It also introduces adaptive memory matching and learned gates to improve how information is stored and retrieved. Short-term and long-term memory are two specialized attention-like mechanisms operating over different parts of the context, while persistent memory is a standard neural network that stores knowledge in its parameters. Memory Mosaics v2 matches Transformers on standard language benchmarks, with both scoring 52.2% on average. It shines at remembering new information from long texts, scoring 53.4% compared to 41.1% for Transformers on 32K-token tasks. It also learns new tasks better from examples, with improvements of more than 10 percentage points on some classification tasks. Even with 8 times more training data, Transformers still fall behind Memory Mosaics on long-document question answering. The goal of this research is to help language models remember new information and learn new tasks from examples more effectively. It focuses on improving how models use information during inference, rather than simply learning more knowledge during training. It is not yet a replacement for Transformers because its main advantages appear in specific memory and learning tasks, while its overall performance on standard benchmarks is similar and its efficiency at large scale is not explored.

📝 DeepLoop: Depth Scaling for Looped Transformers Training very deep Transformers can become unstable because small changes can build up as information passes through many layers. This becomes even harder when the same model components are used many times, because their updates can have a stronger combined effect. Looped Transformers reuse the same layers several times instead of adding new ones. This gives the model more computation without adding many more parameters. DeepLoop shows that this repeated reuse changes how strongly updates can affect the model, so the scaling rule should be adjusted. Residual connections are important because they help information and gradients flow through deep models and important for stability during training. Residual connections carry the current representation forward while the attention or MLP branch adds a new change. DeepLoop changes this balance by making the skip path stronger with α and starting the residual branch with smaller weights using β. Their main contribution is to propose new scaling factors for the residual connection, instead of using the DeepNorm values designed for standard Transformers. In particular, they change the scaling exponent from 1/4 to 1/2 so the model stays more stable when the same layers are reused many times. DeepLoop improves validation loss more as the number of loops increases. It also gives better average accuracy on most downstream settings, including a +3.25 point gain on ARC-AGI at the main voting setting. This suggests the new scaling becomes more useful when layer reuse is stronger.