Ilan Strauss

Memory is becoming a core part of agent harnesses/products (e.g. Instinct, @Muse, @cognition Devin, @PrimeIntellect Prime Agent, @NousResearch Hermes, @openclaw, @LangChain Deep Agents, @Letta_AI) -- yet despite every agent claiming to "continually learn", there's no benchmark to actually measure that capability. A big problem is that memory benchmarks today only measure recall (e.g. simple Q/A about past events). They don't measure if/how agents actually learn from experience to get better at work. At the same time, benchmarks for coding/knowledge work consist of many isolated, unrelated tasks - so they also don't allow for measuring cross-task learning. The context needed for a task is fully specified up-front (very unrealistic). In practice, both agents and humans work through sequences of many *related* tasks. Experience from past work provides context and learnings that apply to future work. This is why if you're a manager, you have to provide a lot more context when assigning tasks to a brand new employee or external contractor. Meanwhile, a great senior employee might not need any instructions at all. With FrontierMemory, we adapt existing benchmark tasks for coding and knowledge work so that instead of all the necessary context being provided up-front and within the task, it has to be learned from past experience. This allows us to compare how well different models and harnesses learn over time.