Bipul Vaibhav (@bv_skillate) profile picture

Building https://nitter.cf/t.co/EYNLqmVv6p | Ex-CEO @Skillate_lab (Acquired by SenseHQ)

San Francisco, CA
Joined August 2010
Photo by Bipul Vaibhav (@bv_skillate)Photo by Bipul Vaibhav (@bv_skillate)Photo by Bipul Vaibhav (@bv_skillate)
Bipul Vaibhav retweeted
So inspiring 🫡
30
136
14
1,066
41,382
I love #DSPy - highly recommended for anyone building an advanced agentic system by far!
1
1
13
2,256
Wow! "Attention is All You Need" (i.e., Transformers) was inspired by the Alien's communication style in the movie Arrival.
Readers added context they thought people might want to know
This is an error. The podcast got this story wrong. Arrival was not an inspiration for self-attention/the transformer, it was however used as an analogy. bsky.app/profile/victor… bsky.app/profile/tilman…
137
566
312
5,095
1,046,102
In the above example, the master agent is PresentationAIAgent, which can build presentations about any topic by getting help from SearchAgent which browses deeply on the web and provides the response.
1
1
312
#DSPy helps build the architecture module by module to keep the agentic architecture structured and cleaner! @AndrewYNg @lateinteraction @tom_doerr
2
309
Agents collaboration must be asynchronous for building a large-scale network to execute complex tasks! #DSPy seems to be a great framework for building that.
3
334
Is anyone working on Nuclear Engineering or has domain knowledge? Please DM me, would love to connect !!
2
600
Tweet of the day, @elonmusk !!
We should send rockets not at each other, but rather to the stars
2
584
Future AGI can be about solving the "binding problem" i.e. stitching together the different sensors like vision, text, and audio into a single model in a balanced way.
1
4
373
Bipul Vaibhav retweeted
# explaining llm.c in layman terms Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity. For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very general Tensor abstraction (a way to arrange and manipulate arrays of numbers that hold the parameters and activations of the neural network), a very general Autograd engine for backpropagation (the algorithm that trains the neural network parameters), and a large collection of deep learning layers you may wish to use in your neural network. The PyTorch project is 3,327,184 lines of code in 11,449 files. On top of that, PyTorch is written in Python, which is itself a very high-level language. You have to run the Python interpreter to translate your training code into low-level computer instructions. For example the cPython project that does this translation is 2,437,955 lines of code across 4,306 files. I am deleting all of this complexity and boiling the LLM training down to its bare essentials, speaking directly to the computer in a very low-level language (C), and with no other library dependencies. The only abstraction below this is the assembly code itself. I think people find it surprising that, by comparison to the above, training an LLM like GPT-2 is actually only a ~1000 lines of code in C in a single file. I am achieving this compression by implementing the neural network training algorithm for GPT-2 directly in C. This is difficult because you have to understand the training algorithm in detail, be able to derive all the forward and backward pass of backpropagation for all the layers, and implement all the array indexing calculations very carefully because you don’t have the PyTorch tensor abstraction available. So it’s a very brittle thing to arrange, but once you do, and you verify the correctness by checking agains PyTorch, you’re left with something very simple, small and imo quite beautiful. Okay so why don’t people do this all the time? Number 1: you are giving up a large amount of flexibility. If you want to change your neural network around, in PyTorch you’d be changing maybe one line of code. In llm.c, the change would most likely touch a lot more code, may be a lot more difficult, and require more expertise. E.g. if it’s a new operation, you may have to do some calculus, and write both its forward pass and backward pass for backpropagation, and make sure it is mathematically correct. Number 2: you are giving up speed, at least initially. There is no fully free lunch - you shouldn’t expect state of the art speed in just 1,000 lines. PyTorch does a lot of work in the background to make sure that the neural network is very efficient. Not only do all the Tensor operations very carefully call the most efficient CUDA kernels, but also there is for example torch.compile, which further analyzes and optimizes your neural network and how it could run on your computer most efficiently. Now, in principle, llm.c should be able to call all the same kernels and do it directly. But this requires some more work and attention, and just like in (1), if you change anything about your neural network or the computer you’re running on, you may have to call different kernels, with different parameters, and you may have to make more changes manually. So TLDR: llm.c is a direct implementation of training GPT-2. This implementation turns out to be surprisingly short. No other neural network is supported, only GPT-2, and if you want to change anything about the network, it requires expertise. Luckily, all state of the art LLMs are actually not a very large departure from GPT-2 at all, so this is not as strong of a constraint as you might think. And llm.c has to be additionally tuned and refined, but in principle I think it should be able to almost match (or even outperform, because we get rid of all the overhead?) PyTorch, with not too much more code than where it is today, for most modern LLMs. And why I am working on it? Because it’s fun. It’s also educational, because those 1,000 lines of very simple C are all that is needed, nothing else. It's just a few arrays of numbers and some simple math operations over their elements like + and *. And it might even turn out to be practically useful with some more work that is ongoing.
This Post is from an account that no longer exists. Learn more
301
1,164
229
9,486
1,801,673
Insightful contrast: !! The fundamental difference is humans mostly learn from the intuition of the physical world whereas even the most advanced LLMs like GPT-4, and LLAMA 2 became intelligent after getting trained on the massive dataset. @ylecun @lexfridman #AGI #LLMs
* Language is low bandwidth: less than 12 bytes/second. A person can read 270 words/minutes, or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token). A modern LLM is typically trained with 1x10^13 two-byte tokens, which is 2x10^13 bytes. This would take about 100,000 years for a person to read (at 12 hours a day). * Vision is much higher bandwidth: about 20MB/s. Each of the two optical nerves has 1 million nerve fibers, each carrying about 10 bytes per second. A 4 year-old child has been awake a total 16,000 hours, which translates into 1x10^15 bytes. In other words: - The data bandwidth of visual perception is roughly 16 million times higher than the data bandwidth of written (or spoken) language. - In a mere 4 years, a child has seen 50 times more data than the biggest LLMs trained on all the text publicly available on the internet. This tells us three things: 1. Yes, text is redundant, and visual signals in the optical nerves are even more redundant (despite being 100x compressed versions of the photoreceptor outputs in the retina). But redundancy in data is *precisely* what we need for Self-Supervised Learning to capture the structure of the data. The more redundancy, the better for SSL. 2. Most of human knowledge (and almost all of animal knowledge) comes from our sensory experience of the physical world. Language is the icing on the cake. We need the cake to support the icing. 3. There is *absolutely no way in hell* we will ever reach human-level AI without getting machines to learn from high-bandwidth sensory inputs, such as vision. Yes, humans can get smart without vision, even pretty smart without vision and audition. But not without touch. Touch is pretty high bandwidth, too.
3
214
Bipul Vaibhav retweeted
* Language is low bandwidth: less than 12 bytes/second. A person can read 270 words/minutes, or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token). A modern LLM is typically trained with 1x10^13 two-byte tokens, which is 2x10^13 bytes. This would take about 100,000 years for a person to read (at 12 hours a day). * Vision is much higher bandwidth: about 20MB/s. Each of the two optical nerves has 1 million nerve fibers, each carrying about 10 bytes per second. A 4 year-old child has been awake a total 16,000 hours, which translates into 1x10^15 bytes. In other words: - The data bandwidth of visual perception is roughly 16 million times higher than the data bandwidth of written (or spoken) language. - In a mere 4 years, a child has seen 50 times more data than the biggest LLMs trained on all the text publicly available on the internet. This tells us three things: 1. Yes, text is redundant, and visual signals in the optical nerves are even more redundant (despite being 100x compressed versions of the photoreceptor outputs in the retina). But redundancy in data is *precisely* what we need for Self-Supervised Learning to capture the structure of the data. The more redundancy, the better for SSL. 2. Most of human knowledge (and almost all of animal knowledge) comes from our sensory experience of the physical world. Language is the icing on the cake. We need the cake to support the icing. 3. There is *absolutely no way in hell* we will ever reach human-level AI without getting machines to learn from high-bandwidth sensory inputs, such as vision. Yes, humans can get smart without vision, even pretty smart without vision and audition. But not without touch. Touch is pretty high bandwidth, too.
This is an essential point people seem to misrepresent.
546
1,640
334
8,446
1,854,368
AI is magical - boils down to Maths and Physics !! Treat to watch when it gets trained very fast and renders beyond imaginable response.
2
112
What a breakthrough in interplanetary communication @NASA !!
We just streamed the first ultra-HD video brought to you via laser from deep space. And it’s a video of Taters, a tabby cat. This test will pave the way for high-data-rate communications in support of the next giant leap: sending humans to Mars. go.nasa.gov/47XDYom
1
120
Need 92 @wasimakramlive like spells from Bumrah/shami now to get 2 consecutive wickets. #INDvsAUSfinal @ICC
356