Khalid خالد retweeted
I put together a handbook on how attention works, from the Q/K/V equation up through how it actually runs in modern LLMs.
Self vs. cross-attention, causal masking, multi-head attention, RoPE, KV caching, MHA/MQA/GQA/MLA, FlashAttention, worked through by hand, with citations back to the original papers.
Made it to be the explanation I wish I'd had.
Khalid خالد retweeted
🚨 New: We built @a16z's personal GPU AI Workstation Founders Edition
- 4x NVIDIA RTX 6000 PRO Blackwell Max-Q (384GB total VRAM)
- 8TB of NVMe PCIe 5.0 storage
- AMD Threadripper PRO 7975WX (32 cores, 64 threads)
- 256GB ECC DDR5 RAM
- 1650Watts at peak (runs on a standard 15Amp/120V circuit).
For training, AI research, and deploying models locally. A datacenter-class AI rig you can keep under your desk.
We are planning to make a limited number of these a16z AI Workstations.
Build guide + how you can make your own 👇
Khalid خالد retweeted
WTF... This scraper works like a self-driving car.
You give it a destination… it plans the route, clicks through pages, cleans it up & delivers a perfect spreadsheet.
Goodbye manual scraping, Excel & data teams.
Tutorial + some crazy examples: 🧵
Khalid خالد retweeted
RLVR/RLHF libraries:
• verl - ByteDance
• TRL - HuggingFace
• slime - Zhipu AI
• prime-rl - Prime Intellect
• ROLL - Alibaba
• Nemo-RL - NVIDIA
• AReaL - Ant Research
• SkyRL - UC Berkeley
• open-instruct - Allen AI
• torchtune - PyTorch
Any I am missing? Which do you like?
Khalid خالد retweeted
time to finally read this one, my Lead Engineer forced me today to read this and to do AoC using C++
I told him I had never done competitive coding so he said go read this and do Advent of Code. Idk what he is cooking but all his advice till now has worked for me
Khalid خالد retweeted
🚨 New Paper Alert 🚨
A Systematic Survey and Critical Review on Evaluating
Large Language Models: Challenges, Limitations, and Recommendations
(1/) VCs are putting thousands of $$$ depending of certain benchmarks while they can be easily hacked or misinterpreted.
(2/) A pretrainers nightmare is not "how to tweak" the model, rather it's "what to tweak" in the model. Without evals it's almost impossible to take educative decision.
(3/) There is a notion in the community that LLM evals are broken, but there's hardly any summary. (👇)
Khalid خالد retweeted
اختبر هوراس هاي GPT-4 مع بعض مسائل البرمجة من موقع يدعى Codeforces. وما الذي جعل موقع Codeforces جيداً حقاً لهذه التجربة الصغيرة بالذات هو أن تاريخ إصدار كل مسألة برمجة مذكور صراحةً. لذا ما فعله هو أنه جمع 10 ألغاز تم إصدارها في الوقت الذي كان يتم فيه تدريب GPT-4 (لذا من المحتمل أن تكون في بيانات التدريب الخاصة به) وقام بسؤال GPT-4 حيث حلها GPT-4 كلها بشكل صحيح.
يا للهول! إنه قادم لوظائفنا.
ولكن ما فعله بعد ذلك هو أنه جمع 10 ألغاز أخرى تم إصدارها بعد تدريب GPT-4. كانت هذه الألغاز بنفس مستوى الصعوبة، ولكن هذه المرة، أخطأ GPT-4 في كل واحدة منها.
إذن ماذا حدث؟ حسناً، أثار هذا الكثير من الشكوك حول أن GPT-4 كان قادراً على حل هذه الألغاز فقط لأنها موجودة في بيانات التدريب وأنه حفظها فقط.
السرد:
nitter.cf/cHHillee/status/…
المحاضرة:
youtu.be/Pv0cfsastFs?si=CY4q…
Khalid خالد retweeted
Happy to share that BLOOM+1 has been accepted to #ACL2023/#acl2023nlp (thanks to all @BigscienceW contributors)! If you want to know how to adapt BLOOM/BLOOMZ to an unseen language, come check out our work!
TL;DR: smaller models benefit from continued pretraining, and larger models from adapters.
(Repost for corrected Arxiv)
🧐What’s the best way to quickly adapt large multilingual language models to new languages?
We present our new paper from @BigscienceW 🌸:
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting.
📜 arxiv.org/abs/2212.09535
[1/9]
Khalid خالد retweeted
11 ways ChatGPT saves me hours of work every day, and why you'll never outcompete those who use AI effectively.
A list for those who write code:
1 of 16
Khalid خالد retweeted
Over the xmas break I was shuffling between stability.ai and JUWELS Booster super computer trying to find some less busy slurm queues to train the first sets of 'ConvNeXt-Base' models. This past week the first Large came off the line on Stability.
Khalid خالد retweeted
Bito is a new #ChatGPT-like assistant for your IDE (@pycharm / VS @Code) or @googlechrome which makes it easy to:
✔️ Write/explain code
✔️ Understand syntax
✔️ Write test cases
✔️ Check security
✔️ Fix bugs
Bito is in alpha & is free to use! 🔥🔥🔥
🔗 bito.co/
للأسف اضافة chatgpt بهذا الشكل هي محاولة لأخذ صدى اعلامي. ولا كان بالإمكان اضافة ملاحظة في abstract أن chatgpt استخدم في بناء وكتابة هذه الورقة.
#ChatGPT بدأ ينشر بحوث شكله بيصير له h-index قريباً
صحيح ما عنده شهادات ولا أساسيات البحث العلمي .. لكن عنده الإمكانيات 😂
1
223
Khalid خالد retweeted
⭐️PhD Interviews⭐️
Now that most PhD app deadlines have passed. Next step would be interviews: Here I had compiled a list of FAQs that was typical from my exp. last year.
Link: docs.google.com/document/d/1…
#PhD #PhDApplications #phdlife #Fall2023
1/n
Khalid خالد retweeted
(Repost for corrected Arxiv)
🧐What’s the best way to quickly adapt large multilingual language models to new languages?
We present our new paper from @BigscienceW 🌸:
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting.
📜 arxiv.org/abs/2212.09535
[1/9]
Khalid خالد retweeted
Wow!
The #ChatGPT Desktop app now supports importing all the prompts from prompts.chat 🔥🔥🔥
Grab it for Windows, macOS, or Linux!
A big shoutout to @fkadev & @lencx_ for their excellent work on developing the app! 🙌
Khalid خالد retweeted
How could you even begin to find theoretical guarantees for zero-shot learning? Our team takes a step in this direction, providing the first non-trivial theoretical guarantees for zero-shot learning with attributes.
Poster #329, Hall J, Thur. Dec. 1, 4-6 PM
[1/3] #NeurIPS22
Khalid خالد retweeted
We are pleased to announce Adawat أدوات, our latest work from @arabicml2 led by @Emad_A_Alghamdi and @superrzk. We create a catalogue for Arabic NLP tools with colab notebooks to test them.
Website: arbml.github.io/adawat
GitHub: github.com/ARBML/adawat
Khalid خالد retweeted
Excited to announce dar a library that can create @huggingface datasets loading script without writing code by answering questions. Using dar I created scripts for +150 Arabic NLP datasets including classification, ner, nli, mt, qa, speech, etc.
GitHub: github.com/ARBML/dar































