jon_chun retweeted
This is a critically flawed paper that sacrifices science at the alter of narrative. It moves silently between fact and speculation, obscuring the inconsistency of its central claim. An autopsy 👇
Let's start with the facts, largely clear by now to those following along:
(1/)🧵
The False Promise of Imitating Proprietary LLMs
Open-sourced LLMs are adept at mimicking ChatGPT’s style but not its factuality. There exists a substantial capabilities gap, which requires better base LM.
arxiv.org/abs/2305.15717
jon_chun retweeted
Ok… So this week:
- Google Bard
- Adobe Firefly
- NVIDIA Foundations
- Bing adds new DALL-E
- Opera adds AI
- Microsoft Loop
- Canva puts all the AIs in
- ChatGPT starts adding plugins
Who’s keeping up? What did I miss?
Anyone else exhausted?
jon_chun retweeted
Bain Capital Ventures just announced $1.9B in new funds
8VC just announced a $880M fund
Felicis just announced a $825M fund
SignalFire just announced a $900M fund
GGV is raising $2.5B in new funds
Khosla Ventures is raising $3B in new funds
etc.
This is not 2001.
jon_chun retweeted
My Twitter timeline is full of panicked takes about imminent AI apocalypse and certain doom.
I think this is starting to get overplayed, and so I want to make a long thread about why I'm personally not worried yet.
Get ready for a big one... 1/n
jon_chun retweeted
So many of the study techniques we learned in school were actually ineffective.
Here are the result of a large meta-analysis of study techniques. You should skip highlighting, summarizing, and rereading. Instead, practice, quiz & explain. journals.sagepub.com/stoken/…
AI can help! 1/
jon_chun retweeted
Very, very interesting: AI without hallucinations is very boring and much less transformative
Bing now lets you chose between creative and precise mode. Here's "find me interesting connections between the theory of diffusion of innovations, Swiss cheese, and US Patent 3,387,396"
jon_chun retweeted
There are AI denialists, AI cautionists, AI boomerists and AI doomerists. I am merely a curios and fascinated AI expectationist.
jon_chun retweeted
What’s the best way to use pre-trained features when you only have a very small training dataset, (e.g. 2 images per class)?
We find that projecting the features onto a good low-dimensional basis can improve generalization a lot.
Paper: arxiv.org/abs/2302.05441
🧵 1/8
jon_chun retweeted
Replying to @nonmayorpete
Google lost $100 b in market cap after the demo failed.
Not making this up
cnbc.com/2023/02/08/alphabet…
jon_chun retweeted
What if you could fit an *entire codebase* in an LLM? 🤔
"Efficiently Scaling Transformer Inference" (11/2022)
arxiv.org/pdf/2211.05102.pdf
Jeff Dean + co break out all the hacks to scale PALM-540B's context length to 43,000 tokens!
Here's how 👇
jon_chun retweeted
I played with @AnthropicAI assistant (AA) and compared it to @OpenAI ChatGPT
TLDR: both are similar but AA is
+ Harder to jailbreak
+ Tries to be more helpful
+ Follows closer what we ask for
+ ~Better for writing in English
- Worst for code
- Worst in French
- Longer resp.
🧵
jon_chun retweeted
Are Flan-T5, T0, and OPT-IML the only open-source, instruction-trained models?
Maybe @AiEleuther has something in the works to compete with OpenAI's InstructGPT models?
jon_chun retweeted
Replying to @PeterHndrsn
The compressed database argument will fail because SD compresses 8exabytes into 8gigabytes, essentially arguing that it stole less than 0.0000001% of each low res image, which is less than 1 pixel. It's a nonsense argument.
jon_chun retweeted
Ok so I downloaded all ~322 episodes of @lexfridman podcast and used OpenAI Whisper to transcribe them. I'm hosting the transcriptions on... "Lexicap" ;) : karpathy.ai/lexicap. Raw vtt transcripts are included for anyone else who'd like to play (they are quite great!)
jon_chun retweeted
I just used 3,000 GPU-hours to test all 9 new OpenAI Whisper speech recognition models, two Talon acoustic models, and NVIDIA Nemo large and xlarge models. Whisper has a peculiar failure case. Here's what I think.
jon_chun retweeted
One way to read the Unnatural Instructions paper is as a clear indictment on human effort. For most knowledge tasks, we can now assume language models are 10x to 100x cheaper than humans. This is striking! As a society, we are unprepared for this tsunami.
arxiv.org/abs/2212.09689

ALT Based on OpenAI’s pricing as of December 2022, the cost for generating an example is estimated at $0.02 for our core dataset, and $0.01 for the expanded dataset. Kiela et al. (2021) estimate human annotation cost at $0.50–$1.00 per example, excluding indirect costs such as task design and UX development; for comparison with our automatic data collection method, we assume the lower-bound human annotation cost of $0.50.
































