Patrick (@patrick_physics) header
Patrick (@patrick_physics) profile picture

physics-to-ml-pipeline:latest

England
Joined January 2009
Photo by Patrick (@patrick_physics)Photo by Patrick (@patrick_physics)Photo by Patrick (@patrick_physics)Photo by Patrick (@patrick_physics)Photo by Patrick (@patrick_physics)Photo by Patrick (@patrick_physics)
me watching oppenheimer
2
120
Who evals the evals?
70
This is huge: Llama-v2 is open source, with a license that authorizes commercial use! This is going to change the landscape of the LLM market. Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers Pretrained and fine-tuned models are available with 7B, 13B and 70B parameters. Llama-2 website: ai.meta.com/llama/ Llama-2 paper: ai.meta.com/research/publica… A number of personalities from industry and academia have endorsed our open source approach: about.fb.com/news/2023/07/ll…
379
3,326
751
14,794
4,294,791
Replying to @typedfemale
model extraction
1
308
BREAKING CHANGE: can we all calm down
Replying to @ryxcommar
What many people fail to recognize is that code is editable
1
99
Replying to @sh_reya @vboykis
tired: attention is all you need wired: xgboost is all you need h/t @tunguz
64
Replying to @sh_reya @vboykis
this, 100%. rumours of the demise of data science are greatly exaggerated
1
137
Replying to @Tim_Dettmers
Thanks a lot for the hard work on the CUDA kernels. I'm curious if batch_size=1 is optimal for this method, or if in the future it may be possible to speed up batched inference at 4 bit also?
1
416
Supersymmetry 2.0
We need new technical breakthroughs to steer and control AI systems much smarter than us. Our new Superalignment team aims to solve this problem within 4 years, and we’re dedicating 20% of the compute we've secured to date towards this problem. Join us! openai.com/blog/introducing-…
83
there is a huge difference between knowledge and expertise. individual humans are not so great with acquiring and retaining knowledge, but are remarkable at developing deep expertise. language models seem to be the exact opposite. they are amazing in the knowledge part.
3
14
83
11,080
*cries in bias-variance tradeoff*
This quoted post is unavailable.
95
Replying to @chrisalbon
AFAIK only important for deep neural networks, so I would keep the card as a good lesson for everything else. Would still love to see a card with an intuitive explanation for double descent
1
1
49
I think it's a horrible idea to ask for licenses to train models, this will reduce number of open-source models and will give big corporations an unfair competitive advantage to train closed-source models which will not be transparent at all and companies will have to sacrifice a lot of privacy since they don't own their models. Monopolies aren't good. If a system affects people, people have the right to know more about it (biases and so on), given we know companies are using these models in all kinds of bad ways (they even use it for candidate screening 😔)
9
14
119
15,322
I get the desire to regulate but placing restrictions on open source orgs is not the right way. Whoever deploys the open source model should be responsible instead
"Any model made available in the EU, without first passing extensive, and expensive, licensing, would subject companies to massive fines of the greater of €20,000,000 or 4% of worldwide revenue. Opensource developers, and hosting services such as GitHub... would be liable"
81
Replying to @yacineMTB
Does this also help get the right answers out of chatGPT?
1
33
Replying to @chrisalbon
Looking for a cause?
20
How is 4-bit fine tuning stable? I guess I will wait and see :)
344
open source is catching up to closed models quicker than expected, although still a few unknowns
Oops haven't tweeted too much recently; I'm mostly watching with interest the open source LLM ecosystem experiencing early signs of a cambrian explosion. Roughly speaking the story as of now: 1. Pretraining LLM base models remains very expensive. Think: supercomputer + months. 2. But finetuning LLMs is turning out to be very cheap and effective due to recent PEFT (parameter efficient training) techniques that work surprisingly well, e.g. LoRA / LLaMA-Adapter, and other awesome work, e.g. low precision as in bitsandbytes library. Think: few GPUs + day, even for very large models. 3. Therefore, the cambrian explosion, which requires wide reach and a lot of experimentation, is quite tractable due to (2), but only conditioned on (1). 4. The de facto OG release of (1) was Facebook's sorry Meta's LLaMA release - a very well executed high quality series of models from 7B all the way to 65B, trained nice and long, correctly ignoring the "Chinchilla trap". But LLaMA weights are research-only, been locked down behind forms, but have also awkwardly leaked all over the place... it's a bit messy. 5. In absence of an available and permissive (1), (2) cannot fully proceed. So there are a number of efforts on (1), under the banner "LLaMA but actually open", with e.g. current models from @togethercompute, @MosaicML ~matching the performance of the smallest (7B) LLaMA model, and @AiEleuther , @StabilityAI nearby. For now, things are moving along (e.g. see the 10 chat finetuned models released last ~week, and projects like llama.cpp and friends) but a bit awkwardly due to LLaMA weights being open but not really but still. And most interestingly, a lot of questions of intuition remain to be resolved, e.g. especially around how well finetuned model work in practice, even at smaller scales.
1
115
This happens if you tell python to f-off too much
Them: "What did you accomplish today?" Me: "Well, the 'f' in the f-string was INSIDE the quotes instead of outside. So that was like... two hours."
83