
Cybersecurity Researcher at Oak Ridge National Laboratory (@ORNLCyber). PhD @UTK_TCE. Applications of AI to cyber, usable security, network security.
Joined January 2021
- Tweets476
- Following41
- Followers46
- Likes67
Sean Oesch retweeted
Turns out LLMs can do some pretty esoteric things in-context!
My favorite paper this week systematically studies LLM beyond natural language. GPT can do complex 2D pattern recognition, fit a sinusoid, predict robot grasp, and even control an inverted pendulum in an online feedback loop.
Below is the extremely challenging ARC IQ benchmark by @fchollet. Without any finetuning, Both text-davinci-003 and PaLM can solve a non-trivial subset (85 and 42 out of 800) after flattening the 2D pattern into a 1D token sequence.
Consistent tokenization is critical. In fact, you can use random tokens sampled from the vocab and still get decent performance.
Next, GPT can predict robot grasp after converting a low-res image into a flat sequence of integer tokens! This suggests some basic computer vision emerging from an extremely powerful text pattern recognizer. Think of it as doing object detection on ASCII art.
Next, GPT can fit & extrapolate a sinusoid function by observing an array of continuous numbers and inferring its amplitude/frequency. This is done purely in-context and without the help of external tools like scikit-learn.
Finally, GPT can control a "left/right" button to balance an inverted pendulum (the classical "Cartpole" task in reinforcement learning) by reacting to the angle & velocity of the pole in an online feedback loop.
There're a lot more experiments & discussions in the paper, "Large Language Models as General Pattern Machines". Take a look!
- Link: arxiv.org/abs/2307.04721
- Authors rom Stanford & Google DeepMind: @suvir_m @xf1280 @peteflorence @brian_ichter @DannyDriess Montserrat Gonzalez Arenas, Kanishka Rao, @DorsaSadigh @andyzeng_
Sean Oesch retweeted
This is way more impactful than it sounds.
Python executes on a single thread regardless of how many CPU cores you have. Removing the notorious GIL (Global Interpreter Lock) means that Python can finally run true concurrency.
Can't wait to see this happen.
Wow. @Meta commits to dedicate three engineer-years to implement the removal of the GIL from #Python and fix upcoming compatibility and performance issues with it.
All this dependent on whether the Steering Council accepts PEP 703.
discuss.python.org/t/a-fast-…
Sean Oesch retweeted
Code is here (including highly valuable prompts): github.com/whitead/paper-qa
Still needs some work, but fun to play around with and has been legitimately useful for my own research. 5/5
Sean Oesch retweeted
A paperclip maximizer, a stochastic parrot, king Midas, a jpeg of the web, markets, democracy, an alien, a shoggoth,...
So many metaphors offered for AI. I discuss these and their limitations.
windowsontheory.org/2023/06/…
Sean Oesch retweeted
It’s fascinating that somewhere in Princeton’s fruit fly brain scan lies Nature’s algorithm of attention, working memory, and even basic consciousness.
Think of the connectome as a compiled executable binary. How much source code can we decompile from a full brain simulation? 🤔
We are releasing a whole-brain connectome of the fruit fly, including ~130k annotated neurons and tens of millions of typed synapses!
Explore the connectome: codex.flywire.ai/
Reconstruction paper: biorxiv.org/content/10.1101/…
Annotation paper: biorxiv.org/content/10.1101/…
1/6
Sean Oesch retweeted
We've all dealt with activation functions while working with neural nets.
- Sigmoid
- Tanh
- ReLu & Leaky ReLu
- Gelu
Ever wondered why they are so important❓🤔
Let me explain it to you in this 🧵👇
Sean Oesch retweeted
GPT-Engineer just hit 12,000 stars on Github.
It's an AI agent that can write an entire codebase with a prompt and learn how you want your code to look.
▸ Asks clarifying questions
▸ Generates technical spec
▸ Writes all necessary code
▸ Easy to add your own reasoning steps, modify, and experiment
▸ Lets you finish a coding project in minutes.
Sean Oesch retweeted
I'd like to have a real conversation about whether AI is a risk for human extinction. Honestly, I don't get how AI poses this risk.
What are your thoughts? And, who do you think has a thoughtful perspective on how AI poses this risk that I should talk to?
Sean Oesch retweeted
In three in one and one in three, in rhyme,
In music, in the whole creation story,
In His own image, His imagination,
The Triune Poet makes us for His glory,
A Sonnet for Trinity Sunday malcolmguite.wordpress.com/2…
Sean Oesch retweeted
1/ Thrilled to announce: Our new course ChatGPT Prompt Engineering for Developers, created together with @OpenAI, is available now for free! Access it here: deeplearning.ai/short-course…
Sean Oesch retweeted
R.I.P. Photoshop.
In just a few clicks, you'll be able to edit any image EXACTLY the way you want. 🤯
Sean Oesch retweeted
🤔Humans can learn from embodied experiences in the physical world. Can Language Models also do that?
🔥Check out our new paper about enhancing Language Models with World Models!
👇arxiv.org/abs/2305.10626
1/n
Sean Oesch retweeted
Generative LLMs are slow and expensive to serve. Their much smaller, distilled versions are faster and cheaper but achieve suboptimal generative performance. We show it is possible to achieve the best of both worlds.
Code: github.com/flexflow/FlexFlow…
Paper: cs.cmu.edu/~zhihaoj2/papers/…
Sean Oesch retweeted
Remember when naïve hackers were called script kiddies?
What would be the equivalent designation for folks who think they are AI experts after a couple hours of LLM prompt engineering?
Prompt kiddies?
Sean Oesch retweeted
Open AI just released their most powerful tool yet.
The Code Interpreter.
Here are 10 insane examples I've found:
Sean Oesch retweeted
What is ignored by the media -- but will be studied by historians?
8 more examples:
Sean Oesch retweeted
1. Hugging Face launches a ChatGPT alternative
Hugging Face launched a 30-billion-parameter open-sourced AI chatbot called HuggingChat
The chatbot aims to compete with OpenAI's ChatGPT.
Sean Oesch retweeted
This paper is going viral.
Why?
Authors were able to find a way to enable the Recurrent Memory Transformer to retain information across up to 2 million tokens 🤯
In simpler words, it can process and remember vast amounts of data, significantly more than before.
Just so you have an idea, GPT-4 handles 32K tokens (~50 pages of documents), while the entire Harry Potter series is ~1.5M tokens.
Imagine the potential implications of this:
• AI could write entire novels, not just blog posts
• It could aid in complex scientific research by analyzing vast quantities of data
• Enhance customer service by retaining years of interaction history
• It could store and recall the entirety of your life experiences!!
Sean Oesch retweeted
I am at least 3-5x more productive using ChatGPT to code.
Not only am I faster writing code I'm familiar with, but I've even shipped apps in tech stacks I'd never used before.
Here's my process, the prompts I use, and why it all works:
Sean Oesch retweeted
My friend saw this book on @malcolmguite's shelf, decided that I needed one on my shelf, then hunted it down and surprised me with it today.
It contains works by favorites such as, Tennyson, Coleridge, Donne, Spenser, Solomon, Yeats.
And it was published the year I was born.





























