Garindra Prahandono retweeted
Replying to @bryan_caplan
both the most powerful tool for learning, and the most powerful tool for not learning
Garindra Prahandono retweeted
The first and only time a business jet was equipped with an afterburner, 1988
some of the best gems on youtube are under 1k views
this one's under 250
youtube.com/watch?v=IBsK1LNM…
119
Garindra Prahandono retweeted
Ditantang 200 km/jam, Pak Basuki Hadimuljono menjawab 210 km/jam. Kami memang suka tantangan.
Garindra Prahandono retweeted
My H-1B experience:
- Worked at a startup, (willingly) earned less than I could have at bigger Silicon Valley companies…
- …but got paid the same as my American coworkers.
Later started a company creating jobs in the US and globally.
Paid millions in taxes over these 14 years. I’d call that a net positive for the US.
A startup paying qualified foreigners (and Americans) a little less because it’s running on fumes isn’t the problem. Outsourcing firms exploiting the H-1B visas are.
Vice President JD Vance breaks down the data:
"If you are working as an H-1B visa in the United States of America, you’re earning $20,000 less than an American citizen hired in the same position. If you were brought in by one of these foreign outsourcing firms, you’re making $48,000 less than an American citizen in a similar position. So I’m sure that everybody here understands the game that is being played. You bring in indentured servants from outside the country, you lay off American workers and, if you’re a corporation, you make a ton of money by under cutting the wages of American workers, replacing them with people who frankly shouldn’t be in the United States of America to begin with."
Garindra Prahandono retweeted
Every person receiving the nation’s top scientific award is a foreign born immigrant
Malaysian GP after 9 years. the cars glisten in the golden hour sun. Opuses humming in the background
could be worse
1
1
227
they'd run the X-59, let it fly by, then.. "oh and one more thing..."
I asked @NASAAdmin about the SR-71 speculation and here was his answer at the Hello America tech summit last night:
“So we are rebuilding our X plane portfolio at NASA. We have X-59 right now, our quiet supersonic transport aircraft or demonstrator. We're hoping to bring it to MAX Power in November. So if you want to get out to an amazing air show, Kennedy Space Center's normally restricted airspace and runway is going to be opened up for everyone to be very exciting, and we are committed to flying as high and as fast as we once did in decades past…”
@NASA @CNBC
118
this is 2007 Steve Jobs
SECRETARY RUBIO on AMERICA.GOV: A United States passport says something very simple. It says that anywhere in the world, "I am an American and my country stands behind me."
160
Garindra Prahandono retweeted
really impressive record by @DevenPzak, still amazed to see new nanogpt speedrun breakthroughs when we thought it was plateauing
they scaled the ngram table from 640M to 65B (2 OOM more), new softmax and optimizer (variant of muon), crazy work
blog: hyperstition.cc/training-nan…
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!
This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.
There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.
The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
github.com/KellerJordan/modd…
As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
Garindra Prahandono retweeted
I find the Chinese names for Indonesian cities so pretty 🇮🇩
Jakarta 雅加达 (Yǎ Jiā Dá) = Elegant + Add + Reach
Surabaya 泗水 (Sì Shuǐ) = Si River
Medan 棉兰 (Mián Lán) = Cotton + Orchid
Banjarmasin 马辰 (Mǎ Chén) = Horse + Dawn
Pontianak 坤甸 (Kūn Diàn) = Earth + Countryside
Bandung 万隆 (Wàn Lóng) = Ten Thousand + Prosperity
Yogyakarta 日惹 (Rì Rě) = Sun + Attract
Palembang 巨港 (Jù Gǎng) = Great + Harbor





























