Alex Booker (@bookercodes) header
Alex Booker (@bookercodes) profile picture

Alex Booker

@bookercodes

Observability and Evals @Mastra

 London
booker.codes
Joined July 2015
  • Tweets7.2K
  • Following654
  • Followers9.6K
949Media
Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)Photo by Alex Booker (@bookercodes)
Alex Booker retweeted
Mastra (@mastra) profile picture
Mastra
@mastra9h
Every vector database has its own SDK, query syntax, and filter rules. Over the last six months we've added seven: same 𝚌𝚛𝚎𝚊𝚝𝚎𝙸𝚗𝚍𝚎𝚡, 𝚞𝚙𝚜𝚎𝚛𝚝, 𝚚𝚞𝚎𝚛𝚢, and 𝚏𝚒𝚕𝚝𝚎𝚛 on Weaviate, Couchbase, OpenSearch, Turbopuffer, etc Swap stores w/o rewriting your code:
5
5
5
41
3,374
We're making the Mastra Observability dashboard best-in-class And everything in it works headless too So you can ask your coding agent to inspect traces and aggregates, and even fix things - potentially automatically 👀
3
4
269
Two latency metrics every agent should track TTFT → how fast your model responds TTFC → how long your user actually waits Fast TTFT but slow TTFC? You probably introduced slow context injection or some other processor
Image from Alex Booker (@bookercodes)
2
7
538
This is on my mind because we just added both to Mastra's Traces table We're adding it to dashboards (follow trends) and experiments next
Image from Alex Booker (@bookercodes)
80
I've spent the last 2 months going deep on observability and evals In next week's workshop I'll skip the cruft and teach you everything I know
Alex Booker (@bookercodes) profile pictureAlex Booker
@bookercodes14h
nitter.cf/i/broadcasts/1aKbdEjyV…
Video
0:00

Alex Booker

Observability and Evals Masterclass

1
3
294
nitter.cf/i/broadcasts/1aKbdEjyV…
Video
0:00

Alex Booker

Observability and Evals Masterclass

1
1
2
402
Right now Mastra traces only supports two statuses. Ok or Error We're adding Suspended, Aborted, Limited, Blocked and more More granular means you can tell at a glance why a run stopped!
Image from Alex Booker (@bookercodes)
1
1
5
335
Observational Memory comes out on top. Again 😏 Here's how it works
Sam Bhagwat (@calcsam) profile pictureSam Bhagwat
@calcsamOct 29
Agent memory gets harder the longer a chat runs. We wondered: how well does Observational Memory hold up at 100K, 1M and 10M tokens? So we ran it on BEAM, the long-term memory benchmark, with default settings. Turns out it has the top published scores at 100K and 1M:
Image from Sam Bhagwat (@calcsam)
1
4
9
509
On a mission to build the cleanest observability dash for Mastra agents Today we added a histogram And input previews - so you see a clean input and not a gross blob of JSON
Image from Alex Booker (@bookercodes)
1
2
13
464
Histogram is @mfrachet and @ShlomesLew's handywork
84
Testing adding a marker for each deployment so you can correlate agent metrics with deploys Would this be helpful?
Image from Alex Booker (@bookercodes)
1
5
284
Mastra Observability is starting to look very comprehensive indeed 🤓☝️
Sam Bhagwat (@calcsam) profile pictureSam Bhagwat
@calcsamOct 28
Today we're excited to announce span queries and span groups in Mastra! We started with trace-level queries; now we're diving to span-level which is more granular AND more performant. Filter/group by time range, span type, name, status, error, duration:
2
13
584
Do you have an agent you want to build for your team? I'm doing a hands-on Mastra intro on Thursday - from TypeScript to a deployed agent in Slack. Join me (and bring ideas!) luma.com/mastra-88zp
Build, Deploy, Observe TypeScript Agents on Mastra · Luma

Build, Deploy, Observe TypeScript Agents on Mastra · Luma

You probably have a list of agents you’d like to build. A new product. A better customer experience. A way to free your team from repetitive work. The harder…

luma.com
1
3
13
603
Alex Booker retweeted
Sam Bhagwat (@calcsam) profile picture
Sam Bhagwat
@calcsamOct 22
Builds are back up. Our recommendation now: revoke and rotate keys for any env variables stored on Mastra platform. To reiterate: we have no evidence of impact.
Sam Bhagwat (@calcsam) profile pictureSam Bhagwat
@calcsamOct 21
Mastra's platform is hosted on Railway and thus Mastra platform users are affected by the ongoing Railway security incident. Comms here as things evolve.
3
3
17
2,022
Alex Booker retweeted
Sam Bhagwat (@calcsam) profile picture
Sam Bhagwat
@calcsamOct 21
Mastra's platform is hosted on Railway and thus Mastra platform users are affected by the ongoing Railway security incident. Comms here as things evolve.
Jake (@JustJake) profile pictureJake
@JustJakeOct 21
Today, we were alerted of malicious activity that compromised Railway's servers We are sharing this with you broadly and swiftly. We recommend folks rotate their keys out of an abundance of caution.
4
6
1
96
14,622
You can now reorder columns in the Traces view Small quality of life improvement
6
476
We redesigned the Mastra trace viewer so you can follow an agent run as a conversation instead of digging through a tree of spans Quick demo with real runs from our docs agent
Sam Bhagwat (@calcsam) profile pictureSam Bhagwat
@calcsamOct 12
Incredibly stoked to launch Thread View for traces in the Mastra studio. Zoom in to see a single chat turn and its associated span. Zoom out to see the whole conversation and underlying trace:
1
14
825
My ambition is for everyone building with Mastra to use Mastra Observability for their traces This week we shipped - Thread View - Metrics API - Trace imports - And now trace queries
Sam Bhagwat (@calcsam) profile pictureSam Bhagwat
@calcsamOct 15
Today we're incredibly excited to launch trace queries! The point of collecting observability data is to analyze it and improve your agent. So now you (or your coding agent) can query by fields like time, primitive name, agent/workflow ID, status, score, and feedback:
2
24
1,219
Live Workshop: Build a Company Brain with Mastra Connect nitter.cf/i/broadcasts/1RJjpbZyd…
Video
0:00

Alex Booker

Live Workshop: Build a Company Brain with Mastra Connect

5
297
Join Charlie and I tomorrow for a workshop on how to build a company brain with Mastra Connect! You'll learn to: - Connect an agent to your knowledge bases - Give it tools to look things up and take action - Put it in your team's Slack 🚀 luma.com/mastra-304z
Build a Company Brain with Mastra Connect · Luma

Build a Company Brain with Mastra Connect · Luma

Your team’s work lives in Notion, Linear, Slack, and other workspace apps. Giving agents access to those tools has meant building integrations yourself or…

luma.com
2
5
398
Load more