CEO at @Letta_AI, prev PhD at UC Berkeley (MemGPT, RL, agents)
SF
Joined March 2014
- Tweets1.9K
- Following1.3K
- Followers4.3K
- Likes6.5K
Pinned Tweet
Continual learning in token space: letta.com/blog/continual-lea…
Charles Packer retweeted
Memory is critical for personal/professional agents, but I suspect that "memory rot' is going to increasingly become an issue.
Basically, agents make bad memory edits that cascade and cause a downward spiral where memory and instructions become worse and worse over time.
The main issue I have with personal agents is the constant drifting from set instructions.
Whether it's Muse or Grok Bot, the start of the experience is amazing.
Proactive updates, always checking everything thoroughly and amazing memory of past chats and how we work.
But over time the bots just enter this autopilot stage where no matter what routine, auditing or micro-managing is done, they start to answer fast rather than researching properly with the established workflow.
The quality of the work dips as well, and I have to spend more time reminding them how to do a task and understanding what changed.
I have instructions in place, I have an entire Notion set up just for my bots to store memory, instructions and ongoing projects and work.
But the proactivity to check this just seems to vanish over time.
My idea was that the chats are too long and there is too much to remember so I gave the bots a place to put info and the instructions to check there before answering something.
But they sometimes just don't.
And I'm struggling to find a solution. Most of my credits on Grok Bot this week have gone to asking them how an email was missed, why the formatting of documents has changed, why the notion isn't updated when it's part of the establushed processes.
If anyone had an idea of how to improve this, would be amazing.
GPT models are significantly worse than the open weights frontier models (Kimi, GLM) at memory generation.
OpenAI's models have a bad tendency to create "rules" as memory, which are the opposite of generalized learnings.
Opus 5.5 leads here - still nowhere near the ceiling.
Charles Packer retweeted
some things I often say to claude when doing perf optimization work:
> how do we rewrite this to better leverage data-oriented design principles
> can you replace those hash tables with a flat array? can we make the pointers an index into the array? can we pack the most commonly used data into upper bits of the index?
> caching is cope for a slow design. do not add caches and delete all existing caches. instead, redesign it to be faster
> should this be a structure of arrays instead of an array of structures?
Charles Packer retweeted
We've updated FrontierMemory to v1.1, with a larger 750-trajectory corpus that reduces reward hacking and includes a more balanced mix of sources.
Opus 5.5 remains the strongest memory model, with open-weight models like DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3 close behind.
You can now view FrontierMemory results inside the greater @FireworksAI_HQ Index, alongside the rankings from Harvey Lab, APEX Agents, FrontierSWE, DeepSWE, and more: fireworks.ai/specialized-int…
FrontierMemory is now on @FireworksAI_HQ's Specialized Intelligence Index.
Picking a model for a long-running agent? You can now compare how well each one learns from its own past attempts, combined with cost.
View FrontierMemory results under RSI: fireworks.ai/specialized-int…
The memory frontier is dominated by open weights models: Opus... then GLM, Kimi, DeepSeek.
The quality gap between the top 3 and bottom 3 in this list is quite wide in practice. Memory generation is far from saturated though, looking forward to more progress in this domain.
We've added GPT-6.1 Terra, Kimi K3.1 Pro and DeepSeek V4.1 Flash to the FrontierMemory leaderboard.
Kimi K3.1 Pro ties Opus 5.5 and GLM-5.3 for #1, and DeepSeek V4.1 Flash beats GPT-6.1 Sol and Gemini 4 Argon, for a fraction of the cost.
Charles Packer retweeted
Caring is the moat, you should feel visceral pain when things are bad or broken and stop at nothing to fix them
Charles Packer retweeted
You can now run FrontierMemory to measure how well your model or harness improves with experience.
We’re also now accepting submissions to the official leaderboard.
Code and submission instructions are available at:
github.com/letta-ai/frontier…
Charles Packer retweeted
Really cool agentic memory benchmark from @Letta_AI . Exciting to see the community thinking more about long-horizon, evolving tasks, and not just static memory recall scenarios.
No company is more invested in open AI than @nvidia, no individual more than @JensenHuang
Incredibly excited to be working the NVIDIA team.. more to come!
Honored to have been a featured ecosystem partner by @JensenHuang at GTC Berlin!
We're excited to partner with @NVIDIA on research towards developing agents that continually learn and self-improve.
Charles Packer retweeted
Memory is one of the biggest unsolved challenges in AI today.
@Letta_AI’s FrontierMemory is the first benchmark to measure AI memory by downstream task performance, across models and harnesses.
Amazing work team @Letta_AI 👏👏
cc @khoslaventures @vkhosla
Charles Packer retweeted
Most memory benchmarks test recall and not synthesis.
My former students just launched an exciting new memory benchmark that evaluates whether agents can continually learn from past experiences.
Charles Packer retweeted
Recommended reading and a super-interesting benchmark for measuring continual learning capabilities through memory.
Agent memory is largely unsolved. As agents advance in capabilities like long-horizon work, it's unclear which models or harnesses work best, or which memory systems and types to use.
Opus 5.5 at the top is not a surprise. But it's exciting that open-weight models like GLM-5.3 perform so well on memory generation.
My big question is how this (better memory systems and persistent agent capabilities) translates into more performant, efficient RSI agents and harnesses.
Charles Packer retweeted
Prime agent #2 on FrontierMemory ranking after Letta Dream!
Great to see Prime Agent do so well on this new self-improving memory benchmark.
My only nit: Let's add agent versions to results!
FrontierMemory also allows measuring the ability of agent harnesses to improve over time. We find that Letta Dream and @PrimeIntellect's Prime Agent improve the most from learning from prior experience data.
