Developer Relations at Qdrant ๐
Seattle, WA
Joined November 2017
- Tweets637
- Following590
- Followers283
- Likes1.2K
kanungle ๐งโ๐ retweeted
Introducing ๐๐๐ฎ๐๐ฒ๐บ๐ฏ๐ฒ๐ฑ, a static embedding inference library written in Rust๐ฆ (with python bindings!๐): github.com/AstraBert/statembโฆ
I've always been a fan of static embeddings: they're fast and lightweight, and they can easily run on CPU, so I wanted to learn how they work.
What better way for learning than implementing inference myself? Starting from a safetensor model and its tokenizer, I wrote the code to load the tensor from bytes, map it to token embeddings, tokenize text and do lookups.
Using potion-8M for benchmarks, statembed turned out to be 3-7x faster than model2vec in rust and 2.5-7x faster in python. The tricks for faster inference are mostly mmap loading for the tensor, using tokenizers V1, and eager loading of the token-to-embedding map so that inference is just a lookup, with no slicing of the tensor and decoding the bytes to floats on the fly (although also lazy loading is supported for larger models).
See the full benchmark results: astrabert.github.io/statembeโฆ
And, if you try it out and have any feedback, please let me know!โจ
kanungle ๐งโ๐ retweeted
This assistant remembers where you left your keys. No network connection. No remote server.ย
Build one yourself in a course built in partnership with Qdrant and taught by Dylan Couzon.ย
๐ Enroll for free: hubs.la/Q04zGYrF0
#AI #VectorSearch #AIEngineering
kanungle ๐งโ๐ retweeted
Build an assistant that can store and search what it sees, hears, and is told, all without leaving the device.
In Building AI Assistants with On-Device Memory, you'll build a local memory system that recalls text, voice, and images, and teach it to recognize something new from a few photos.
Built in partnership with @qdrant_engine and taught by @DylanCouzon, Developer Experience Engineer at Qdrant.
This course also includes our new AI coding lab, so you can practice building with an AI coding agent from the DeepLearning.AI mobile app.
Enroll for free: hubs.la/Q04y3S0F0
kanungle ๐งโ๐ retweeted
77ร less vector RAM. 94.5% of baseline retrieval quality.
We got early access to Google EmbeddingGemma 2 and tested how far its Matryoshka-compatible embeddings could be compressed in Qdrant.
With shorter vectors + quantization + rescoring, we reduced the vector memory footprint by ~77ร while retaining 94.5% of baseline retrieval quality.
Google also featured our work in their EmbeddingGemma 2 announcement.
Qdrant deep dive: qdrant.tech/blog/embeddinggeโฆ
Googleโs announcement: blog.google/innovation-and-aโฆ
kanungle ๐งโ๐ retweeted
What happens when engineers are given a controversial topic and told to debate it?
Weโre bringing something a little different to San Francisco Tech Week.
Join Qdrant ร @neo4j for Hard Negatives: Engineers Debate Night on October 9.
Weโll put engineers, AI researchers, and builders into 1v1 debates on spicy topics like:
โOpen weights should be banned.โ
The audience votes live on who wins each round.
Plus, your RSVP includes food and one drink.
October 9 | 6:30โ9:00 PM PDT
Spots are limited and registration requires approval.
Link: luma.com/sf-meetup-oct26
This is a game-changer, and maybe the first of its kind.
Asymmetric Dual Encoders allow you to use a normal "heavier" embedding model to encode your documents, while then giving you a "hot-swappable" model for query embeddings.
You can use 400M-parameter Stella model for you documents, but on the query side choose from Zero, Nano, or Stella itself (see image for latencies).
"Constella-Zero" is a learned bag of tokens that operates via look up table, pooling, and normalization. The results are shockingly good.
"Constella-Nano" is a transformer model with 1/12th the parameters of Stella, and even better results
Average nDCG@10 across 15 datasets:
-> Zero = 0.4572
-> Nano = 0.5081
-> Stella = 0.5614
Read more about the training process, results, and getting started resources below:
qdrant.tech/blog/constella-rโฆ
Credit to @DylanCouzon for the research work and @generall931 for the idea
Note: Currently in "research preview"
I think our leaders, especially political leaders (dem and rep) are stuck in a 'famine' mindset. It's slowing us from reaching our potential as a species.
Whether it's attacking immigrants, AI data centers, industrialists, or just pure corruption -- it's all unhelpful.
We're on the cusp of nearly free-energy powered data centers in space, abundant AI compute, abundant resources, and the ability to do it with full sustainability.
Today, yes, there is waste. There are misallocated resources. Backward investments and unsustainable practices persist.
But it doesn't have to be this way.
Jevon's paradox is already showing us that we'll increasingly need human labor in nearly all the fields AI is touching today. As we push towards our destiny as a multiplanetary species, there will be no shortage of the need for humans.
I envision us working closely together with AI and robotics for a long, long time, as we explore new frontiers.
We must do it the right way, though.
Despite the noise and the criticism, I'm inspired to see people working to push us all forward, with a clear vision of a better future.
I, for one, am excited to see @Google's launch of #Suncatcher orbital data center next week, and the 14th launch of @spacexai's #starship with new @Starlink satellites.
We're pushing foward, whether you acknowledge it or not. Don't be stuck in the past. Don't be stuck in the 'famine' mentality.
Come, as you are
Nirvana's Nevermind turns 35 today
See where it falls on our list of the best albums of the 1990s: pitchfork.visitlink.me/TCkYxโฆ
for the jev-inclined search engineers out there
Cheating at Search problems with Jev
Jev shows impressive promise for query understanding. Though hierarchical classification remains challenging
softwaredoug.com/blog/2026/0โฆ
400k+ documents moved.
Similar latency
Less Memory
kaivid.com/blogs/migrate-milโฆ
kanungle ๐งโ๐ retweeted
I'm talking about Inference Free SPLADE models at @softwaredoug's Retrieval Augmented Gathering today.
SPLADE uses a transformer neural network to expand your document's keywords, so keyword search can match synonyms (and other semantically similar keywords). But it runs a model on every query, which adds upto 50ms p50 latency on CPU and needs a GPU to be faster.
Inference-free SPLADE skips that step. It does more model work upfront on your documents, so queries cost about as much as BM25. We'll talk about how it works, what you trade off, and how to take it to production :)
Join today if you wanna learn more! ๐บ๐
Miss me? Watch me for 4 hours straight here! LOL ๐ฎโ๐จ
Vector Space Stream | Latest Research from Qdrant nitter.cf/i/broadcasts/1PKqrNwdOโฆ
Vector Space Stream (Live R&D mega-session) kicks off online in just a few hours. Join us!
luma.com/vector-space-stream
