Manos Tsagkias (@samanos) profile picture

Siri@Apple. X-Founder @904labs, @SolumbroLeisure, @MyYardSoftware

San Francisco
Joined June 2009
Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)Photo by Manos Tsagkias (@samanos)
Manos Tsagkias retweeted
Today in Nature: our research on world and human action models (WHAM) - generative ai models of video games, aimed towards supporting game creatives in gameplay ideation : nature.com/articles/s41586-0… - huge congrats to everyone who made this happen, I couldn't be more proud 🥳
20
56
8
217
28,498
Manos Tsagkias retweeted
Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n
25
386
77
1,913
468,636
Manos Tsagkias retweeted
"What Matters In Transformers?" is an interesting paper (arxiv.org/abs/2406.15786) that finds you can actually remove half of the attention layers in LLMs like Llama without noticeably reducing modeling performance. The concept is relatively simple. The authors delete attention layers, MLP layers, or entire transformer blocks: - Removing entire transformer blocks leads to significant performance degradation. - Removing MLP layers results in significant performance degradation. - Removing attention layers causes almost no performance degradation! In Llama 2 70B, even if half of the attention layers are deleted (which results in a 48% speed-up), there's only a 2.4% decrease in the model benchmarks. The author also recently added Llama 3 results to the paper, which are similar. The attention layers were not removed randomly but based on a cosine-based similarity score: If the input and output are very similar, the layer is redundant and can be removed. This is a super intriguing result and could potentially be combined with various model compression techniques (like pruning and quantization) for compounding effects. Furthermore, the layers are removed in a one-shot fashion (versus iterative fashion), and no (re)training is required after the removal. However, retraining the model after the removal could potentially even recover some of the lost performance. Overall, a very simple but very interesting study. It appears there might be lots of computational redundancy in larger architectures. One big caveat of this study, though, is that the focus is mostly on academic benchmarks (HellaSwag, MMLU, etc.). It's unclear how well the models perform on benchmarks measuring conversational performance.
72
491
36
2,909
195,574
Live at @SMLions with Dr. William Dunham discussing Leonhard Euler, the "Master of Us All". He was almost exact contemporary with Benjamin Frankin
1
2
1
12
New reqs for low to high level researcher positions: jobs.careers.microsoft.com/g… , jobs.careers.microsoft.com/g…, jobs.careers.microsoft.com/g…, jobs.careers.microsoft.com/g…, with postdocs from Akshay and @MiroDudik nitter.cf/MiroDudik/status/18367… . Please apply or pass to those who may :-)
31
2
107
36,172
Manos Tsagkias retweeted
This is the nerdiest article on org design I have ever read. Inject it straight into my brain! codahale.com//work-is-work/
7
22
6
232
26,296
Manos Tsagkias retweeted
Just in time for the holidays, we are releasing some new software today from Apple machine learning research. MLX is an efficient machine learning framework specifically designed for Apple silicon (i.e. your laptop!) Code: github.com/ml-explore/mlx Docs: ml-explore.github.io/mlx/bui…
95
672
135
4,377
962,015
Code search seems an interesting area for IR w/ a mix of challenges from (micro-)blog and desktop search: rapid updates, and little internal structure (e.g., links). Do we have datasets for code search? theregister.com/2023/02/07/g…
2
168
Still going through the #SIGIR2022 proceedings. Here is another gem: a neat workflow of which metric to choose for which ranking problem. From "Ranking Interruptus: When Truncated Rankings Are Better and How to Measure That" by Enrico Amigó et al. PDF: damianospina.com/publication…
3
13
60
The misconception that there is no sound in space originates because most space is a ~vacuum, providing no way for sound waves to travel. A galaxy cluster has so much gas that we've picked up actual sound. Here it's amplified, and mixed with other data, to hear a black hole!
11,373
102,144
33,642
411,435
The times are a-changin’? Google for general search, and TikTok for lifestyle search? A more elaborate version of this thread would make an interesting IR paper.
As an elder millennial who came of age when Google became a verb, I had difficulty grasping that ~40% of Gen Zers use TikTok for search. So I hosted a mini-focus group with Gen Zers to learn more. Here are 3 takeaways:
2
Manos Tsagkias retweeted
1) What is LaMDA and What Does it Want? cajundiscordian.medium.com/w… 2) Interview cajundiscordian.medium.com/i… What can be said with confidence imo is that things are about to get a lot weirder because models appear to follow smooth scaling laws and data+model size can still plenty grow.
109
400
64
3,031
We've built THE list of VC lists. 23 databases. 220k+ investors. 100% free. Please like + RT this tweet to help a founder out. Aaaand... here we go 👇
116
1,266
138
4,380
Just discovered Hedy, a gradual programming language. It's a new idea on how to teach programming to beginners. Very cool! hedycode.com/ @hedycode
14
124
14
599
Manos Tsagkias retweeted
Most of the #trec2021 baselines on the MS MARCO V2 data are now available in Anserini as reproducible regression tests. Enjoy! github.com/castorini/anserin…
5
30
Manos Tsagkias retweeted
Proud to share the webpage for the brand-new Microsoft Research Lab Amsterdam where we will work on molecular simulation for green technology and healthcare. We are hiring deep learning for computational chemistry researchers and engineers. microsoft.com/en-us/research…
9
71
4
483
Manos Tsagkias retweeted
An early draft of the machine learning interviews book is out 🥳 The book is open-sourced and free. Job search is a stressful process, and I hope that this effort can help in some way. Contributions and feedback are appreciated! huyenchip.com/ml-interviews-…
44
522
41
2,506
Replying to @karpathy
We've built an openly available, massive text dataset with 800GB: pile.eleuther.ai/ We're also building a massive image and image-text dataset that are open with each image having either CC0, CC-BY or CC-BY-SA license. I'm truly excited for the future of open datasets!
6
61
6
393
Happy to share the preprint for our #icassp2021 paper "Error-driven Pruning of Language Models for Virtual Assistants" w/ the Siri Speech team at @Apple – arxiv.org/abs/2102.07219
1
10