Suprith retweeted
We add 100-150 new designs into the app every day right now.
We use human Creators to work with our model to make designs so it’s human + AI, not just AI to curate content. Our goal will be to increase the quality of design over time so nobody has to use bad designs.
Suprith retweeted
Want to learn how high earners pay less in taxes?
I recorded a 3-hour course breaking down 25 different strategies
We sold the course for $299 last year, but have made it free for the next 24 hours only
RT or comment 💰 and I'll DM you a link
Suprith retweeted
📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
youtu.be/l8pRSuU81PU
The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.
Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
github.com/karpathy/build-na…
Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo
Suprith retweeted
EXCLUSIVE FIRST LOOK: Silicon Valley garage startup building a humanoid robot.
$8,000.
BOM on robot.
@kscalelabs .
3D printed carbon fiber.
26 high torque electric motors that were stuck in customs for a month.
Pre launch.
Will be first shown at @ycombinator demo day in April.
Company didn’t exist three months ago.
This is the purest form of the startup drug.
Suprith retweeted
New YouTube video: 1hr general-audience introduction to Large Language Models
youtube.com/watch?v=zjkBMFhN…
Based on a 30min talk I gave recently; It tries to be non-technical intro, covers mental models for LLM inference, training, finetuning, the emerging LLM OS and LLM Security.
gm L2 enjoyooors
arbitrum.foundation/eligibil…
Suprith retweeted
ESPN is now advertising The Last Dance as "coming soon" opposed to airing in June. Full story on @awfulannouncing momentarily.
Suprith retweeted
Some #Redbirds fly high enough they end up in space. Physics and computational physics double major Zachary Temple was one of 50 students who spent his semester interning at @NASA's Langley Research Center (LaRC) in Hampton, Virginia. https://nitter.cf/t.co/jTClAgKQMy #RedbirdScholar
Suprith retweeted
The #TwitchCon hackathon project fair has begun!
Tune in to see what these developers have accomplished with Twitch Extensions in just 24 hours.
https://nitter.cf/t.co/YV30oPhj0Q
Suprith retweeted
Just Announced: Finish Line Grants $2,500 to help promising hackathons happen. All you need to do: attend peer groups and help other organizers. #HackconVI #keynote
Suprith retweeted
“I’m going to trade for all the shitty contracts on your team so I can sign a max deal this summer with the Lakers but don’t get too excited because that second max is for Kawhi and your ass is going to San Antonio”
Suprith retweeted
When you just landed in fortnite and your teammate says they’re getting shot at – at Bronx, NY
Suprith retweeted
4/18/07: Mark Buehrle threw the 16th no-hitter in White Sox history in a 6-0 victory over Texas. #MB56 nitter.cf/whitesox/status/878251…























