IonelRoiban retweeted
Absolutely classic @GoogleResearch paper on In-Context-Learning by LLMs.
Shows the mechanisms of how LLMs learn in context from examples in the prompt, can pick up new patterns while answering, yet their stored weights never change.
💡The mechanism they reveal for in-context-learning.
When the model reads a few examples in your prompt, it figures out a pattern (like a small rule or function). Instead of permanently changing its stored weights, it forms a temporary adjustment that captures this pattern. That adjustment can be written mathematically as a rank-1 matrix, meaning it only adds one simple direction of change to the existing weights.
This rank-1 update is “low-rank”, so it is very cheap and compact. But it still lets the model shift its behavior to fit the examples in the prompt. Once the prompt is gone, that temporary rank-1 tweak also disappears.
So, in simple terms:
The paper shows that in-context learning happens because the model internally applies a temporary rank-1 (very simple) weight update based on your examples, instead of permanently retraining itself.
---
That behavior looks impossible if learning always means gradient descent.
The authors ask whether the transformer’s own math hides an update inside the forward pass.
They show, each prompt token writes a rank 1 tweak onto the first weight matrix during the forward pass, turning the context into a temporary patch that steers the model like a 1‑step finetune.
Because that patch vanishes after the pass, the stored weights stay frozen, yet the model still adapts to the new pattern carried by the prompt.
---
Shows that the attention part can take what it found in your prompt and package it into a tiny “instruction” that, for this 1 forward pass, acts exactly like a small temporary change to the MLP’s weights.
Nothing is saved to disk, yet the block behaves as if the MLP just got a low-rank tweak computed from your examples. Remove the prompt, the tweak disappears, the saved weights stay the same.
As the model reads your examples token by token, it keeps refining that temporary tweak. Each new token nudges the MLP a bit more toward the rule implied by your examples, similar to taking small gradient steps, again only for this pass.
When the examples have done their job, those nudges shrink toward 0, which is what you want when the pattern has been “locked in” for the current answer.
🧵 Read on 👇
IonelRoiban retweeted
President Trump has approved our request to deploy the #USNSMercy to Los Angeles. Grateful for the quick response and partnership, @realDonaldTrump.
IonelRoiban retweeted
Very good discussion with @realDonaldTrump. In response to the COVID-19 crisis, we are preparing with other countries a new strong initiative in the coming days.
Want to host a live show with multiple guests? You should check out @streamyardapp (StreamYard)! streamyard.com?pal=606155747…
1
IonelRoiban retweeted
Thank you Mr. President!
My friend (always there when I’ve needed him!), Senator @RandPaul, was just tested “positive” from the Chinese Virus. That is not good! He is strong and will get better. Just spoke to him and he was in good spirits.
IonelRoiban retweeted
HYDROXYCHLOROQUINE & AZITHROMYCIN, taken together, have a real chance to be one of the biggest game changers in the history of medicine. The FDA has moved mountains - Thank You! Hopefully they will BOTH (H works better with A, International Journal of Antimicrobial Agents).....
IonelRoiban retweeted
Places where Joe Biden controlled or had influence over US policy:
Ukraine
China
Romania
Places where Hunter Biden did business with corrupt individuals:
Ukraine
China
Romania
It's almost like he followed his dad around the world peddling his last name for power & money
🤔
IonelRoiban retweeted
"The United States of America doesn't stand with murderous dictatorships, we stand up to murderous dictatorships" @VP on #NorthKorea #CPAC2018 #CPAC #ATimeForAction
.@VP on North Korea
Powered by SnappyTV
Four years of initial coin offerings, visualized in one graphic elementus.io/token-sales-his… via @elementus_io
IonelRoiban retweeted
Third burn successful. Exceeded Mars orbit and kept going to the Asteroid Belt.
IonelRoiban retweeted
Live Views of Starman pscp.tv/w/bUjQEDFyYVFab25sTU…
IonelRoiban retweeted
North Korean Leader Kim Jong Un just stated that the “Nuclear Button is on his desk at all times.” Will someone from his depleted and food starved regime please inform him that I too have a Nuclear Button, but it is a much bigger & more powerful one than his, and my Button works!
IonelRoiban retweeted
What a year it’s been, and we're just getting started. Together, we are MAKING AMERICA GREAT AGAIN! Happy New Year!!
IonelRoiban retweeted
One of the most powerful moments of my life watching @POTUS give the Purple Heart to this American Hero. Amazed by the strength and resilience of the men and women in our Armed Forces.
IonelRoiban retweeted
Today, @POTUS signed a directive calling for the United States to lead a human return to the Moon, followed by missions to Mars and beyond. Details: go.nasa.gov/2jOXj5g






















