kaffu (@bythyag) header
kaffu (@bythyag) profile picture

kaffu

@bythyag

joyship // eigenvalue enjoyer

 42
bythyag.bearblog.dev
Joined May 2025
  • Tweets797
  • Following599
  • Followers571
36Media
Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)Photo by kaffu (@bythyag)
new larp
Image from kaffu (@bythyag)
2
6
232
paper link: arxiv.org/pdf/2603.05413
23
told my mom i am editing my resume: le mom: if u quit job this time, you better come back home and prepare for upsc
3
24
429
i can cook multiple cuisine and it is just 5 flavors of maggie
3
24
364
claude giving my sad soul some validation
Image from kaffu (@bythyag)
10
283
Rishta naya rabba dil chu raha hai Kheeche mujhe koi dore, teri ore Teri ore, teri ore, teri ore, hai rabba Teri ore, teri ore, teri ore
2
95
doctor should prescribe me medication for vitamin her and miss her
5
114
niche music taste and it is just prateek kuhad
2
1
15
341
had coffee and croissant and felt very french coded
1
16
300
gm today is sundayyyy
Image from kaffu (@bythyag)
14
243
kaffu retweeted
kaffu (@bythyag) profile picture
kaffu@bythyagOct 24
wrote a technical blog on how to train nanogpt2 in moe style without a load balancer. do give it a read. happy sunday you all! now i should log out, rest and sip some matcha.
Image from kaffu (@bythyag)
3
12
29
598
wrote a technical blog on how to train nanogpt2 in moe style without a load balancer. do give it a read. happy sunday you all! now i should log out, rest and sip some matcha.
Image from kaffu (@bythyag)
3
12
29
598
bythyag.bearblog.dev/removin…
removing the balancers - training nanogpt-2 in moe

removing the balancers - training nanogpt-2 in moe

Abstract Every mixture-of-experts (MoE) model ships with a load balancer. The idea of load balancer is to prevent dead experts - a condition where a few e...

bythyag.bearblog.dev
7
85
just one last experiment bro..just one more last experiment.
2
184
small work update: - trained nanogpt mixture-of-expert without load balancer - matched the same performance with nanogpt dense layer on public benchmark - beats the performance of moe with sota load balancer it was a nice saturday :D
17
352
begged opus 5.5 for some encouragement to feel better about myself:
Image from kaffu (@bythyag)
1
5
173
larping in dms be like:
Image from kaffu (@bythyag)
13
332
in some random 3pm, i am in search of more natural way to cluster than k-means to prevent expert collapse in moe
18
339
having breakdown back to back since last 4 days due to pile up of both work and personal project. i am so bored of building agent harness now and more i work on it, more i realise the anti-thesis of it.
1
8
247
lately discussing with opus/fable on ideas that i want to experiment with. despite me knowing that the approach can have a possible exploration, these ANT mfs would start pushing me back to a different direction i don't really care about. pacing frontier my ass.
12
225
Load more