Amazing to see Russians soldiers using unmanaged US satellites for their own comms usage. Great Youtube post here: youtube.com/watch?v=EUuQwPAP… .
75
Anthony Russell retweeted
Launch and return are fundamental techniques for Starship’s fully and rapidly reusable design
Anthony Russell retweeted
Intel just dropped the most powerful 7B model ever!
- Top 7B on Huggingface's leaderboard.
TL;DR:
- Base model: Mistral 7B.
- Fine Tuned on: SlimOrca
- DPO: LLaMA-13B vs ChatGPT Gens (Prefer ChatGPT)
---
- Model: huggingface.co/Intel/neural-…
- SFT Dataset: huggingface.co/datasets/Open…
- DPO Dataset: huggingface.co/datasets/Inte…
- Code: github.com/intel/intel-exten…
- License: Apache 2
---
---
Unique Hardware:
Trained on Habana Labs's 8x Gaudi2 mezzanine accelerators!
I didn't get to try the accelerators myself yet.
To the best of my knowledge | Hardware TL;DR:
- Comparible in performance to A100.
- 96GB on device RAM.
Read More: habana.ai/habana-labs-data-d…
---
Base model
Mistral 7B [1]
For those not familiar, mistral is considered to be among the best 7B base models nowadays.
> Future deep post about mistral it coming soon.
[1] Paper arxiv.org/abs/2310.06825
---
Supervised Fine Tuning (SFT)
Trained on SlimOrca [1][2]
- SlimOrca is a subset selection of Orca that only includes ~500k GPT-4 completions.
- The selection was done using GPT-4 to remove answers which appear wrong based on the orinigal FLAN targets.
[1] Orca Paper: arxiv.org/abs/2306.02707
[2] SlimOrca Dataset: huggingface.co/datasets/Open…
---
Direct Policy Opimization (DPO)
TL;DR DPO [1]: Given a prompt and a pair of completions, train the model to prefer one over the other.
We currently witness that most of the most powerful models over the past month were trained with DPO.
This model was trained on prompts from SlimOrca's dataset where each has one GPT-4 completion and one LLaMA-13B completion.
The model trained to prefer GPT-4 over LLaMA-13B.
[1] DPO Paper: arxiv.org/pdf/2305.18290.pdf
---
Intel's Codebase
To be honest:
I was skeptical when I found out about Intel's claims.
Usually new hardware is not trivial to work with.
But I must give credit where credit is due.
Intel fully support Huggingface's trainer for all stages of modern LLM training:
- Causal Language Modeling
- Supervised Fine Tuning
- Reinforcement Learning From Human Feedback (Both PPO and DPO)
- And LoRA.
Everything has easy and Minimalistic (thank you!!!) code examples and you can run whatever you want with a push of a button.
The performance is state-of-the-art as written above.
I will try to reproduce their whole pipeline experiment end-2-end over the next few days and report back to you.
---
In Addition!
Habana's hardware is developed in Israel! 🇮🇱
Anthony Russell retweeted
last week we shared how to borrow against real property as an NFT
this week is a video demo on how you can go from paper deed to NFT in a couple of minutes of work
📜 --> 🏡
check it out
Anthony Russell retweeted
Question for the experts 🙋♂️
Will the extraordinary energy requirements of the newest generative AI models help slow their progress, and potentially alleviate safety concerns related to runaway capability?
Cc: @Scobleizer @elonmusk @ESYudkowsky @ylecun @karpathy @amasad
Anthony Russell retweeted
The last decade has been a pivotal one in the AI revolution 🤖
Cc: @ylecun @Scobleizer @erikbryn @amcafee @paulg @DavidSacks
Anthony Russell retweeted
AEMO projecting a collapse in residential and commercial gas demand in eastern Australia- falling by 65% by 2040
This is because electrification now far cheaper and more efficient than gas for cooking, heating and hot water. #auspol
aemo.com.au/-/media/files/ga…

ALT AEMO GSOO 2022, Residential and commercial gas demand, Step Change Scenario, Figure 5, https://aemo.com.au/en/energy-systems/gas/gas-forecasting-and-planning/gas-statement-of-opportunities-gsoo Step Change scenario now overwhelmingly considered the most likely: https://aemo.com.au/newsroom/media-release/nem-prepares-for-step-change#:~:text=Since%20September%202020%2C%20AEMO%20has,the%20most%20likely%20future%20scenario.
Ad spending per vehicle sold:
Genesis: $1,955
Lincoln: $1,553
Jaguar: $1,235
Lexus: $984
Cadillac: $908
Honda: $183
Dodge: $78
Mitsubishi: $43
Tesla: $0
Source: Ad Age (2020 spending)
inside.com/business?utm_sour…
Wonderful to see the crowd at the @wallabies V @FranceRugby in Brisbane all be COVID aware and wearing masks. Again all the COViD concerns are from returning Australians. Not those in country.
1
Great post in @smh on failure of Australian government relating to returning Australians smh.com.au/world/europe/what…
So we’ll written Jason Wilson writing in an opinion piece in @guardian - When I look at Australia from the US, I see an insular nation that has turned its back on its own theguardian.com/commentisfre…
Could not come sooner - let’s hope we see more governments following quick on the NHS’s heals on this. theguardian.com/society/2021…
2
Time to geek out and go down a rabbit hole…. @elonmusk and Sandy Munro …… missed this earlier in the year but very interesting. youtu.be/YAtLTLiqNwg
Could not have said it better @WSJ - The World Relies on One Chip Maker in Taiwan, Leaving Everyone Vulnerable
wsj.com/articles/the-world-r…
1
Great to see Australia even talking about risk based assessments with Covid. 8:45 into this @abcnews interview. Hopefully they will start allowing more people from Singapore in soon. youtu.be/6PXJpy378Zw
Anthony Russell retweeted
Incredible photo of this piece of paradise this winter, shared by the North Bondi RSL. Happy Weekend 😍🐳🌊
We need to consider allowing more immigration from our surrounding neigbours. After living in Asia for many years and seeing the professional nature of the Filipino nurses we should consider allowing more into Australia smh.com.au/national/it-s-hor…












