Abhishek Thakur

Thread of things I've built over the last few years. Most of them started because I wanted to understand how something actually worked instead of treating it as a black box. I'll keep adding to this thread whenever I finish something new.

ngl, it's interesting how an entire llm architecture can be described in a config.json. spent time understanding what these parameters actually mean. now it's time to turn that understanding into my own inference engine in rust.

ngl, i underestimated how much i needed to understand candle before building enma. even basic stuff like creating tensors on cuda, converting to fp16 and reading results back to rust is worth figuring out before i start implementing the actual model

built an inference gateway around llama.cpp, but wanted to know how an inference engine actually works. so i started studying qwen2.5 1.5b and made this architecture diagram myself. ngl, connecting all these parts is frying my brain. wish me luck.

could've jumped straight into coding the inference engine in rust, but decided to understand qwen first. turns out drawing the architecture myself is helping a lot. still have more to figure out, but at least i know what i'm going to implement and why.

i was reading about qwen2.5 and noticed something odd. the tokenizer has 151,665 entries, but the embedding table has 151,936 rows. those 271 extra rows make the table divisible by 128, allowing even splits across different gpu counts.

Bars.

i know these aren't crazy numbers, but damn. people are actually taking the time to read my stuff kudos to you :)

there is so much to understand when you actually start learning how an llm inference engine works every time i understand one concept, five more things show up ahhhhhhhhh

added a request queue to llmgate to handle concurrent requests better learn learn learn .

while working on llmgate, i ran into a problem with usage tracking

i keep coming back to go, rust, and python when i think about building systems. wrote an article on where i'd use each, where this approach falls short, and why ai might make the whole language debate matter less. therealshek.medium.com/go-ru…

Started writing an article, let's see where it goes. :)

slowly adding more to the llmgate arsenal. currently working on metrics and adding prometheus and grafana support. taking it one piece at a time.