
Product Engineer @writewithmarker. Previously CTO @Leap_Labs
Joined July 2008
- Tweets666
- Following756
- Followers384
- Likes966
Robbie McCorkell retweeted
We've been building Disco for two years, to help science escape confirmation bias. Systematic, data-driven discovery for all mankind. Multiple novel findings published. Completely free for public data! And as of today, no waitlist: leap-labs.com/
The thesis: hypothesis-driven science can only find what you already suspect. Confirmation bias, publication bias, path dependence in the literature – most published findings don't replicate.
LLMs trained on papers inherit all of this. Agents can automate what a human would do – hypotheses, analysis code, lit review – but they share all our biases. They search under the streetlights, just like we do. Disco gives them a capability they can't replicate with prompting + pandas.
Because it starts from the data. Neural networks find patterns. Interpretability makes them legible. Hold-out validation and contextualisation make them defensible.
Available via web UI, Python SDK, MCP server, or REST API.
github.com/leap-laboratories…
Robbie McCorkell retweeted
Big new review!
🟦Open Problems in Mechanistic Interpretability🟦
We bring together perspectives from ~30 top researchers to outline the current frontiers of mech interp.
It highlights the open problems that we think the field should prioritize! 🧵
It baffles me that the major cloud providers don't have good solutions for serverless GPU compute yet. I don't want to have to manage VMs or K8s to get access to an A100.
Really impressed with @modal_labs so far. Immediate access to H100 level, no approval, pay per second.
1
9
281
"it also highlighted the need for us as an industry to move past artificial tests to more realistic evaluations that can accurately assess models true capabilities and limitations."
Love this. We need tools that give a holistic view of the model.
Fun story from our internal testing on Claude 3 Opus. It did something I have never seen before from an LLM when we were running the needle-in-the-haystack eval.
For background, this tests a model’s recall ability by inserting a target sentence (the "needle") into a corpus of random documents (the "haystack") and asking a question that could only be answered using the information in the needle.
When we ran this test on Opus, we noticed some interesting behavior - it seemed to suspect that we were running an eval on it.
Here was one of its outputs when we asked Opus to answer a question about pizza toppings by finding a needle within a haystack of a random collection of documents:
Here is the most relevant sentence in the documents:
"The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association."
However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings.
Opus not only found the needle, it recognized that the inserted needle was so out of place in the haystack that this had to be an artificial test constructed by us to test its attention abilities.
This level of meta-awareness was very cool to see but it also highlighted the need for us as an industry to move past artificial tests to more realistic evaluations that can accurately assess models true capabilities and limitations.
1
2
311
Ok tech twitter. I'm doing some Python and want to know the best tools and libraries in the ecosystem.
Ruff looks like a great example. Modern, fast, well documented, takes the best of similar projects for other languages etc.
What else pushes the language forward to be better?
1
195
pdm looks like an interesting (but perhaps early) solution to the mayhem that is pip.
Pydantic looks like a nice package for data validation that could help with my zod addiction in JS land...
2
131
Almost as soon as I tweeted this, @astral_sh once again answered my prayers and released uv for package management. Looks great! github.com/astral-sh/uv
60
What helps you catch bugs early? What helps your team work together better? What encourages you to better structure your code?
1
98
I had a great time talking about MLOps Community: LLMs Mini Summit on the @MLOps community Meetup, you can check out the whole presentation here: home.mlops.community/home/vi…
Robbie McCorkell retweeted
Unlock the secrets of maximizing language models' potential at the MLOps Community Mini Summit #3 brought to us by @wandb!
Robbie McCorkell retweeted
Looking forward to exploring our hidden biases (I mean our model's hidden biases 😉) with @JessicaRumbelow & @robbiemccorkell from @LeapLabs on Sat. 10/28: RSVP-ASAP to secure your spot: Aiify_2023-10-28.eventbrite.… — Details: aiify.io/events/231028sf/
Next on our little Bay Area tour, I'll be showing off the Leap Dashboard at Aiify.io on Saturday
Don't miss this Saturday's Aiify.io – IRL & Livesteam. RSVP ASAP to secure a spot: Aiify_2023-10-28.eventbrite.… — Details: aiify.io/events/231028sf/ Full Schedule and Speaker Roster are live.
Thanks for to those that came to AI Tinkerers SF and saw my quick talk on better AI Interpretability.
Some quick links for those interested: 🧵
Replying to @AlexReibman
9/ Leap Labs
Better AI Interpretability
Demystifying models by identifying which features vision models pay attention most to
@robbiemccorkell
- 🔔 Join our discord for discussion and news on upcoming releases (including LLM support):
discord.gg/scmNtrp6XC
1
80
- 🎓 Our research paper on Image Classification Prototypes:
arxiv.org/abs/2309.17144
57
Robbie McCorkell retweeted
Can we derive robust insights about learned class features directly from a trained model? We propose Prototype Generation, a data-independent feature visualisation method that preserves class features and produces natural internal activations. Paper: arxiv.org/pdf/2309.17144.pdf (1/5)
Robbie McCorkell retweeted
And another two!
Kim Antoniou will share how @auristech is using AI to improve literacy and English language skills for students.
And @robbiemccorkell from @Leap_Labs will teach us how to figure out what image classifiers have learned!
This is gonna be good...
I'm excited to share the next two AI London Speakers!
We'll have the wonderful @pursuitofprog (of Bully XL fame) sharing with us how he rapidly impacted Government Policy (AI Policy, anyone?)
And the fantastic @genmon sharing the state-of-the-art of LLM agent collaboration!


















