Kate Johnson (@KateAthmer) header
Kate Johnson (@KateAthmer) profile picture

Pretty much have Twitter just to reach various hard to reach customer support entities.

Philadelphia, PA
Joined January 2009
Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)Photo by Kate Johnson (@KateAthmer)
Kate Johnson retweeted
We need humans, or human proxies, in reinforcement learning and evaluation loops now more than ever. Agents keep topping benchmarks, then stumbling in the real world. No surprise -- for one, there’s a level of scientific rigor needed to ensure benchmarks don’t test models with tasks they’ve seen in training data. For another, tasks and rubrics included in benchmarks are usually much simpler than what real users hold as good. Finally, tasks and answer keys are often wrong (about 6.5% of MMLU questions have errors, Terminal Bench was recently found to contain errors). How many legal documents can be reviewed without back-and-forth with a client? How many tax returns can be prepared without input from specialists, departments, and regulatory websites? To handle real-world workflows, agents need to learn the real-world messiness during training. That often means integrating humans into training and evaluation environments. But people don’t scale, so teams turn to LLMs to play the user. New research from Meta shows why that’s a problem. The MIMESIS paper measured how often the same GPT-5.5 agent succeeds: With real users: 63.6% success. With frontier-model playing real users: 82.4–84.4%. Too tidy, helpful, and nice. With pretrained Qwen3.5-4B/9B playing real users: Success drops by 16.0 and 14.9 points. Too obstructive. The authors trained simulators on how humans actually interact, closing the gap. Every trained simulator landed within 10.3 points of real users; every off-the-shelf model missed by at least 14.9. The best, MIMESIS-4B and 9B, came within 2.4 points. The key takeaway: trained simulators captured how people actually behave, their directness, emotions, and context, while exposing agents to more diverse interaction paths. The team then added a coaching loop, using the simulator’s private reasoning and next reaction as feedback for the agent. Agents trained this way outperformed those trained against GPT-5.5 across all nine unseen user simulators. Kudos to Hoang Phan and the MIMESIS team at @Meta. This is rigorous, practical work for the industry Our read: The next leap in agents comes from understanding how people really behave. They’re vague, impatient, and change their minds mid-sentence. That’s where we are spending time at Dscout, and we’re happy to help anyone experimenting here. What does your “user” look like when you test your agents? #AIAgents #HumanAI #UserResearch
1
1
1
243
In grad school, my marketing professor wouldn't let me reposition Pedialyte as a hangover cure. "They would never do that, they are about baby safety". I think about him often.... gobbledy.substack.com/p/pedi…
28
Replying to @mada299
My favorite line: use attribution as a diagnostic, not to set goals. The moment a number becomes a target, people start optimizing the number and that's when people start pointing fingers too !
2
1
1
236
Kate Johnson retweeted
My whole CMO career I felt marketing was shortchanged when it came to attribution and now I finally have the data to prove it 😆 🙃 My findings: - On every model, enhancing the data gave marketing more credit. Last touch went from 15% to 25%. First touch from 38% to 45%. Linear multi-touch from 20% to 37%. - CRM/marketing automation touchpoint only see 51% of what creates a deal. The other 49% lives in unstructured data - Brand and advocacy start 28% of deals. The opportunity source field credits them with 5.6%. - Unresponded cold outbound one-way emails fill 45% of logged touches but earn 3.7% of the credit. - AEs only add 2.7 people to buying group, the real group is 7.9 The whole report: upside.tech/research/the-unt…
5
5
2
13
712
But I still have to PRINT out a paper form and MAIL it in to request to set up auto pay on my @FifthThird mortgage account? 🤦‍♀️🤦‍♀️ PRIORITIES PEOPLE
#DarkMode is here! 🌑 Download the Fifth Third mobile app, unless you're afraid of the dark. #Awooooooo
1
77
7-0 🤝 7-0
260
5,475
388
42,629
.@AmericanAir I cannot apply flight credits online, I have to call. When I call, your agent cannot access the same rate I can book online via AA website. You took away the option to hold flights, but the agent keeps asking us to put flights on hold so she can try to book. HELPPP
2
How exactly are we supposed to use flight credits? And why do my husband and I have to book separately to use flight OR trip credits?
.@AmericanAir I’m having trouble getting reimbursed for expenses incurred due to the ACY to PHL bus delay last week (no driver). Gate agent said we’d be reimbursed for our own transportation to PHL. Submitted the customer support form but got an unhelpful reply. Can you help?
1
.@AmericanAir shoutout to the two female flight attendants working main cabin on flight 528 phx to phl right now. Lots of people who don’t know “how to airplane” and they are handling every twist with grace.
1
.@ReadyRefresh we ended our subscription due to a move. Have 5 empty bottles to return. Will you pick them up or how do we recycle? Support never responded.
1
.@Walmart @walmarthelp are you all selling "ghost bikes"? I've ordered this bike twice, and each time it gets "returned to sender - problem with delivery address". The other (more $$) bike on the same order delivered to the same address just fine. 🤨 walmart.com/ip/Kent-Bicycles…
1
Kate Johnson retweeted
Curious about intent data as part of your B2B outreach strategy? What are the best practices around deploying intent data? How are others leveraging intent data in your industry? B2B Fusion teamed with @KateAthmer and Marissa Bialick of Bombora on crea…lnkd.in/dFJnS8jR
1
2
Kate Athmer - @KateAthmer - Harnessing the Power of Intent Data sirkinresearch.com/harnessin…
1
2
.@AmericanAir trying to pay with a flight credit per instructions here -aa.com/i18n/customer-service…, yet such a button does not exist at checkout. Halllppp?
1
Hey @PHLAirport - just called the number on your screens to report a distressed, barefoot, robed lady yelling at a window in baggage C -and was informed that number is only for open investigations. 🤨 (yes I tried to find on site security first but no luck)
1
Robed= wearing what looked like a robe. Red color.
.@philly311 can you help with the truck that's currently stuck and blocking traffic at the corner of 19th and Hamilton (10 mins and counting), because they ignored the "no trucks sign"? PS - the driver is wearing Cowboys gear, what an embarrassment. #dallassucks
.@philly311 are you broken?
Replying to @philly311
@philly311 there's been a huge pothole with a cone in it on Vine street at the corner of 17th for monthssss.
Ugh, sorry @Templerowing
This Post is from an account that no longer exists. Learn more