Albert Lai

There is a CS / stats professor at Berkeley called Michael I. Jordan who has an h index of 200. So the Michael Jordan of machine learning is a guy named Michael Jordan.

Real talk. Vibecoding is fun, and you should try it. But vibemodeling has been one of the most educational and intellectually satisfying things I've done in awhile. All kinds concepts and statistical properties you hear about suddenly become much clearer.

walz confirmed Dreamcast owner. hugely energising for the narrowest voter segment in world history

This is really a 'WOW' paper. 🤯 Claims that MatMul operations can be completely eliminated from LLMs while maintaining strong performance at billion-parameter scales and by utilizing an optimized kernel during inference, their model’s memory consumption can be reduced by more than 10× compared to unoptimized models. 🤯 'Scalable MatMul-free Language Modeling' Concludes that it is possible to create the first scalable MatMul-free LLM that achieves performance on par with state-of-the-art Transformers at billion-parameter scales. 📌 The proposed MatMul-free LLM replaces MatMul operations in dense layers with ternary accumulations using weights constrained to {-1, 0, +1}. This reduces computational cost and memory utilization while preserving network expressiveness. 📌 To remove MatMul from self-attention, the Gated Recurrent Unit (GRU) is optimized to rely solely on element-wise products, creating the MatMul-free Linear GRU (MLGRU) token mixer. The MLGRU simplifies the GRU by removing hidden-state related weights, enabling parallel computation, and replacing remaining weights with ternary matrices. 📌 For MatMul-free channel mixing, the Gated Linear Unit (GLU) is adapted to use BitLinear layers with ternary weights, eliminating expensive MatMuls while maintaining effectiveness in mixing information across channels. 📌 The paper introduces a hardware-efficient fused BitLinear layer that optimizes RMSNorm and BitLinear operations. By fusing these operations and utilizing shared memory, training speed improves by 25.6% and memory consumption reduces by 61% over an unoptimized baseline. 📌 Experimental results show that the MatMul-free LLM achieves competitive performance compared to Transformer++ baselines on downstream tasks, with the performance gap narrowing as model size increases. The scaling law projections suggest MatMul-free LLM can outperform Transformer++ in efficiency and potentially in loss when scaled up. 📌 A custom FPGA accelerator is built to exploit the lightweight operations of the MatMul-free LLM. The accelerator processes billion-parameter scale models at 13W beyond human-readable throughput, demonstrating the potential for brain-like efficiency in future lightweight LLMs.

Every day someone in California is like “went on a walk” and then proceeds to post the most breathtaking landscapes god has to offer

Your order was delivered… to the Moon! 📦 @Int_Machines' uncrewed lunar lander landed at 6:23pm ET (2323 UTC), bringing NASA science to the Moon's surface. These instruments will prepare us for future human exploration of the Moon under #Artemis.

On a wide variety of text, math, coding and reasoning benchmarks, including MMLU, GSM8K, MATH, Big-Bench Hard, HumanEval, Natural2Code, DROP, and WMT23, Gemini outperforms all other models and improves on the state-of-the-art. On MMLU, Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%.

Goodnight Pac-12 football You were so loved. For every bonkers call made by a Pac-12 ref For every INSANE After Dark game that exploded in a chaos no slumbering East coast writer would see For 100+ years of in-state rivalries like Washington vs Wazzu or U$C vs Integrity RIP

arxiv.org/abs/2307.16892 #lk99