built akshara, a bpe tokenizer tuned for languages
at 200k vocab vs gpt-4o's 0200k:
- overall 5.03 vs 4.64 bytes/token
- beats it on all 19 languages tested (wiki)
- biggest gains: thai +52%, japanese +50%, bengali +32%, tamil +27%
- code: ahead on 20 of 21, small gains
- english ~tie (+1%)
compression only, not Im quality
akshara-tokenizer.vercel.app
pov: you are using secondary monitor with your mac and screen sharing is nightmare to change for multiple screens/windows..
building an app which will solve it!