Creator of Firn: vector + full-text search on object storage. Platform & data infra engineer. AWS, Terraform, Flink. AWS Solutions Architect.
Cork, Ireland
Joined March 2007
- Tweets13.7K
- Following1.1K
- Followers1.4K
- Likes5.2K
Pinned Tweet
Firn has a Python package now!
Just `pip install firn`
Vector and full-text search on object storage, straight from Python. The engine embeds in-process, so there's no server: point it at a local folder or an S3 bucket.
Imagine how happy your SysOps team will be with no cluster to manage. Or your CFO, with no managed solution to approve
Ooh, that doesn’t bode well for my experiments using Jev as a reference to try to develop a better BM25 ranking formula 😅
Although I was scoring query/document pairs rather than comparing documents directly. Might be worth testing for position bias in my results
Gordon Murray retweeted
The chances of being killed by a landmine delivered by a bear are low, but they are never zero.
I gave Paperclip and Codex a research challenge: find a better ranking algorithm than BM25.
We tried mathematical formulas, AI distillation, neural models and GPU training.
Some promising results. Plenty of failures too.
Even Jev wasn't better on every dataset.
I wrote up what I learned about search relevance and autonomous AI research gordonmurray.ie/data/2026/10…
🤖 Made with AI
I've been trying to get AI agents to discover a better search ranking formula than BM25.
After testing 432 conventional configurations, I compared the best against Jev on 100 unseen queries.
FiQA nDCG@10:
Tuned BM25+: 0.232
Jev reranking: 0.402
A 73% improvement.
Still haven't found a cheap formula that comes close.
😭
In lieu of flowers, memorial contributions may be made to:
Johns Hopkins Alzheimer’s Disease Research Center Fund
alzresearch.org/giving/
Alzheimer’s Foundation of America Caregiver Resources
alzfdn.org/donate/
Photo Credit: Tommy Garcia
One interesting finding from these experiments
For a scientific query about T-cell activation, BM25 ranked a relevant document 66th.
Jev moved it to 1st.
My experimental formula ranked it 67th 😅
I changed the approach. Instead of asking AI agents to invent ranking formulas, I trained a simple formula using 126,436 ranking preferences from Jev
Results (nDCG@10):
BM25: 0.753
New formula: 0.778
Jev: 0.850
The formula runs without AI inference.
Still early, but interesting. Next step is unseen datasets.
I’ve been experimenting with using AI agents to discover a better ranking formula than BM25, the algorithm used by Elasticsearch and OpenSearch.
One candidate looked promising:
BM25 nDCG@10: 0.753
Experimental: 0.800
Then I tested it against 300 unseen queries:
BM25: 0.660
Experimental: 0.616
Back to the drawing board
I’ve been experimenting with using AI agents to discover a better ranking formula than BM25, the algorithm used by Elasticsearch and OpenSearch.
One candidate looked promising:
BM25 nDCG@10: 0.753
Experimental: 0.800
Then I tested it against 300 unseen queries:
BM25: 0.660
Experimental: 0.616
Back to the drawing board
QuerySpigot now scores 92/100 (A) on Agenstry's A2A conformance check.
Signed agent card, A2A 1.0 interface, ~35 ms anonymous responses, and 1,754 of 1,754 uptime probes up.
Irish company data for AI agents, paid per request in USDC over x402.
queryspigot.com
Very cool
We've added RDS MySQL Blue/Green support to @ProxySQL 3.0.10. You can route test traffic to the replacement database before switching over in AWS.
Thanks, Wazir Ahmed, for the feature and the guide. Have a read:
proxysql.com/blog/proxysql-3…
Gordon Murray retweeted
We should be more ambitious with the software we build:
danluu.com/ambitious/
![Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader, this money overflow bug, etc. This isn't to say that pgrust i...](https://pbs.twimg.com/media/HVbOGdKboAAAbtO.png?name=orig)
ALT Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader, this money overflow bug, etc. This isn't to say that pgrust i...
Nice to see another local lakehouse build. I explored a streaming version with MySQL CDC → Flink → Iceberg, plus Trino, Soda checks and Superset, using Docker Compose.
github.com/gordonmurray/apac…
my new weekend project : local data lakehouse
using:
Apache Spark
Apache Iceberg
MinIO
Trino
Docker Compose
Streamlit
Iceberg REST Catalog
🔗 GitHub: github.com/mohamedmeqlad99/d…
linkedin: linkedin.com/feed/update/urn…
I used this stack for MetaBare: CLIP for images, MiniLM for notes, with BM25 alongside both. Search and embeddings run in the browser; your files stay on your device. The site is static S3 + CloudFront, no expensive GPUs
github.com/gordonmurray/meta…
If you’re using skills for data engineering, I’ve shared a collection for Flink, Iceberg, Paimon, Fluss, Lance and more. Each includes what to inspect first and how to verify the work.
github.com/gordonmurray/data…
We're donating Agent Skills to the Agentic AI Foundation (@AgenticAIFdn), the same foundation we donated MCP to.
Since December 2025, 70+ agent platforms have integrated the standard, including ChatGPT, Cursor, GitHub Copilot, and VS Code.
Here's what that means for you:
I tested Jev with Firn on SciFact: nDCG@10 rose from 0.727 to 0.800. It worked, but I left reranking optional rather than add a hosted dependency. Relevance is only part of the decision.
gordonmurray.ie/data/2026/09…
I hope tpuf does not go this way - I think a big part of the success is that you have access to all the knobs if you want them. The space is moving faster than any one vendor.
e.g. Jev already dethroned on reranking (twice).
hevmind.com/writing/perplexi…
I benchmarked bbtree while considering it for Firn. Integer inserts were ~4.7x faster than BTreeMap, but deletes were slower. The operation mix matters. My tests and caveats:
gordonmurray.ie/data/2026/10…
