Parth Parmar

Fire 🔥

haha fair, simple version: AI models read text in small chunks called tokens. fewer tokens for the same text = cheaper, faster, and more fits in memory most tokenizers are built for english, so hindi, tamil, thai etc get chopped into way more pieces akshara chops them into fewer pieces

built akshara, a bpe tokenizer tuned for languages at 200k vocab vs gpt-4o's 0200k: - overall 5.03 vs 4.64 bytes/token - beats it on all 19 languages tested (wiki) - biggest gains: thai +52%, japanese +50%, bengali +32%, tamil +27% - code: ahead on 20 of 21, small gains - english ~tie (+1%) compression only, not Im quality akshara-tokenizer.vercel.app

make sure you've got your UK visa sorted

giving away a @solana breakpoint ticket share your best project, product or github below 👀 solana.com/breakpoint

Yup

updated my X profile

Where?

Inn

the non-commoditized part of the game

Noice

Chad @shivangpmr

Pen tool only

coming to mumbai next week for solana summit and devcon. I'd like to do some chit-chat around there.