Building AI Products at @Sentry
San Francisco, CA
Joined November 2012
- Tweets882
- Following350
- Followers298
- Likes3.5K
Dhrumil retweeted
We’re also hiring engineers to help build our eval platform and enable every product team at Sentry to ship AI features with confidence: sentry.io/careers/95d2eeab-2…
We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform.
Will share more in a future blog post, but some stats from our devs in the past month:
- ~1800 eval runs triggered directly on PRs
- 1500 eval scenarios ran across 17 AI surfaces in our product
- Cost/token reductions of ~20-50% across a few of our surfaces
- Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev)
All this from essentially 0 back when we started this effort in May.
.@sentry is a great tool to monitor the health of your products. But I don't remember the last time I've actually opened the website.
I just have sentry cli installed, and every morning an agent reads through the issues and flags P1s and P0s for me.
💯 You will encounter ppl that consider themselves to be "high agency" and think that's license to be dismissive or a jerk vs actually making things net positive
> get introduced to someone new
> being told that they are high agency
> ask if they mean breaking the rules and soiling the commons or figure out a way to push the pareto and create positive sum
> she doesn't understand: "they're high agency sir"
> meet the person
> they're soiling the commons and externalizing internalities
Agent Plugin @sentry + Sending traces/logs/errors =
Coding Agents get limitless debuggability
I’ve also seen counts and rates sent as Errors. I have screamed a little inside
Send those as application metrics. Every plan includes 5 GB of metrics per month too and we don't charge for cardinality so add that user_id without worries
And be picky with captureException().
If a third-party API returns an expected, handled 401, track how often it happens as a metric. Keep it as an Error when it breaks a user flow or needs someone to fix it. Your Issues list will thank you.
Want help cleaning this up?
Use the Agent plugin and ask your coding agent to audit your Sentry calls and move routine signals to logs or metrics + build dashboards, while keeping real bugs as Issues.
docs.sentry.io/ai/agent-plug…
Dhrumil retweeted
a weekly cron-job that has an agent pull sentry issues, fix them and create a PR is such an unlock
Spend your time ensuring your agents don’t write crap code
Don’t waste it to query data or debug stuff - agents like Seer are good at it
Everyone keeps asking "how do we actually use Seer?"
Cool. We'll just show you.
Introducing #SeerChats ➡️ real, unfiltered convos between us and our own agent, straight from the Sentry trenches. No cherry-picked demos, no marketing gloss. Just Seer doing what it does (and sometimes what it doesn't) so you can see how it actually solves problems, before you throw your own bugs at it 🫡
In this #SeerChat: docs LCP p95 was spiking, so we asked Seer to find out why.
It compared spike vs. calm traces, found the typical user was hit (not just outliers), and traced it to slow RSC responses stacking with heavy JS work during soft navigations — then handed back a prioritized fix.
You should install the @sentry agent plugin.
Your agents can then use all your telemetry contextually and make better decisions while planning etc
Trying to reduce my token costs and codex automatically picked up live costs from Sentry
docs.sentry.io/ai/agent-plug…
Built a small app for my sister.
Today she used it for real and @sentry told me it was broken before she texted me.
That “knowing before your user tells you” moment never gets old.
There’s no excuse not to do this. Everyone’s apps should be fast. But I would install @sentry and use tracing for better e2e wins.
Dhrumil retweeted
Let's dive into this together! luma.com/agentic-loops-with-… @sergical ❤️
