Associate Professor, UC San Diego #HCI (Human-Computer Interaction), #AI On sabbatical in the Bay Area. Alum. @UToronto, @Tsinghua_Uni
University of California, SD
Joined May 2014
- Tweets285
- Following431
- Followers4.3K
- Likes684
Pinned Tweet
We're building Aware to explore the future of personal computing. What we believe is that when software doesn't work the way you want, you should be able to change it.
Here's another example. Over the years, I've noticed two types of user behaviors:
1. People who save and bookmark many things, but rarely look back.
2. People who don't save anything because they know they won't look back.
The same underlying pain point: likes and bookmarks are siloed across websites and platforms, and they are hard to find later.
So the first thing I built for myself in Aware was simple:
Save anything I bookmark or like across different websites in one place, organize by my own preferences, and let me search through it all.
Aware builds the two connected pieces:
1. an extension-like tool that detects a click on a like, save, or bookmark button on any web page
2. a bookmark space that receives the liked items and their metadata, and I define how they are organized and searched.
We live in a vast digital universe with billions of websites, services, and applications. Yet, we’ve always been visitors, subscribers, and users,
and never the * owners *.
What if we can create our own information universe, one that holds our own data, understands us, and evolves with us?
We have been building Aware, a united place for the internet, your materials, and your agents. This clip shows how you can turn a tab into your *own* tab.
The Aware that I posted earlier can't, but now it can, because it's a malleable environment that you can shape. A simple prompt in Aware and a couple of minutes of agent work, and there you have it.
My prompt to Aware: give me a pop-up to write down my notes when I bookmark on X, and also allow me to search the note in the Bookmarks.
Haijun Xia retweeted
Aware is (basically) a browser we're building to explore personal computing and generative, malleable software. it keeps a footprint of the materials (pages, files) as a graph in a relational database stored locally.
Been writing my paper in my own latex editor inside Aware. It's very satisfying when software fixes itself and fetches its own context and attach them to the right parts.
The editor is an experiment of "interface as a skill": the editor's code and its skill live in the paper's repo, so the same agent that knows my code and data also lives in the editor. So i get:
1. an interface i can reprogram any part anytime (aesthetics i like, and the scratch pad i always wanted)
2. an agent living in the editor, with the whole project as context (a figure links to the dashboard, built from the real analysis code that made it)
(though i have to admit, some of my focused writing time now goes to wanting to tweak the editor instead)
We live in a vast digital universe with billions of websites, services, and applications. Yet, we’ve always been visitors, subscribers, and users,
and never the * owners *.
What if we can create our own information universe, one that holds our own data, understands us, and evolves with us?
We have been building Aware, a united place for the internet, your materials, and your agents. This clip shows how you can turn a tab into your *own* tab.
We live in a vast digital universe with billions of websites, services, and applications. Yet, we’ve always been visitors, subscribers, and users,
and never the * owners *.
What if we can create our own information universe, one that holds our own data, understands us, and evolves with us?
We have been building Aware, a united place for the internet, your materials, and your agents. This clip shows how you can turn a tab into your *own* tab.
Had a blast discussing Dynamic Task UIs with Simra @its_sshahid from @adaption_ai .
If you missed it, this article I published today
nitter.cf/HaijunXia/status/21133…
summarizes my view of the emerging interface paradigm with AI agents.
We spend a lot of time making models more capable, and far less on how people experience and shape that intelligence.
It’s central to Adaptive Interfaces at @adaption_ai. I’m excited to unpack this with @HaijunXia who has been rethinking the future of interfaces!
How should we approach generative user interface systematically?
Rima @YiningCao3 and I have been following the approach of identifying and designing design patterns. for tasks and interfaces. Chatlet addresses the task pattern of collaborative decision making. A Chatlet in a group chat gathers the context and generates the suitable UIs for the group to make decisions in a more visually informed way. Check it out!
Group decisions happen through conversations, but keeping track of everyone's ideas, preferences, and options isn't always easy.
We've been exploring the idea of Chatlet - an interface takes shape and keeps evolving alongside your conversations. You keep your natural conversations with your friends and colleagues (NOT an AI!), while Chatlet evolves to help everyone think, coordinate, and make decisions together.
We believe generative UIs shouldn't have to start from an explicit prompt. Instead, they could emerge from context: understanding what's happening, recognizing when an interface might be useful, and figuring out what form it should take.
This research project is in collaboration with Nathalie Henry Riche, Siân Lindley, and Hugo Romat from Microsoft Research @MSFTResearch , and @HaijunXia from UC San Diego.
We're now looking for early users to try Chatlet, we'd love your feedback!
🔗 Sign up to try Chatlet: chatlet-signup.fly.dev/
👀 Full demo and learn more: rrrima.me/blog/chatlet
I have always loved well-made explainer videos. They are engaging and educational, but used to take days and weeks to make...
With this #skill, every question can be answered with an interactive video and a scrolly essay. Super cool skill, give it a try!
A fun side project that helps with my hundred thousand whys.
One question → an illustrated scrolly essay + a video you can stop anytime to ask your own question. It answers with a new clip that fits into the video.
This came out of a question I keep returning to: which digital things do we actually want to be malleable? I love scrolly essays and explainer videos, but they never quite answer all my whys. So that's the one I wanted to bend. Had a lot of fun.
more examples: explorables.fly.dev
open source, as a Claude Skill: github.com/Rrrima/explorable…
--
voice-over: @ElevenLabs
music & sound effects: @StabilityAI
everything else: Claude Opus 5.5
--
question for this video: How to measure the IQ and EQ of dogs and cats? And how do the breeds rank?
AI makes us create more and do more, but think less and worse. Almost want to start a company with the name of “Thinking Human Lab”.
I’ve seen at least 6 AI try-on + shopping posts this weekend on this platform.
I wonder if that's a wrong use case of image generation?
An image model is designed to make a generated image look plausible and coherent.
A real try-on is to see what’s wrong with this fit.
Too tight at the shoulders or bad length?
If AI smooths over those imperfections to make the image coherent, it’s deleting the exact signal the shoppers are looking for?
We may look great in the images, but not so great in reality?
i’ve spent a ton of hours and tokens prompting @Muse to make me this custom fashion app but holy crap is it worth it:
- gave it my height, weight, measurements, and photos
- gave it my top 20 clothing stores and budget ranges
- it picks a “hero piece” from the catalog
- then builds a look around it (inventing other common pieces)
- i described everything i like and dislike and i keep iterating each time
- these are my “loves” saved category
Eric's messages may exaggerate a bit and be a bit misleading, but I agree with the direction he is taking: that traditional Graphical UI (GUI) is on the decline.
But UIs are not going away, it how we interact with both the physical and digital worlds. As long as there is information transmission, there is a UI.
If we consider user interface modalities broadly- visual/graphical, language, gestural, etc-we are clearly going through a shift from heavy reliance on GUI to language-driven UIs. No single modality will take over or entirely go away, but the redistribution is real, and GUI is on the decline.
Within GUI, WIMP (btw, P stands for pointer or pointing devices, not pull down as Eric said) UI is being impacted the most.
We need to think about non-traditional, non-WIMP GUIs. Most of the human-agent interfaces, even though generated, are still a continuation of WIMP, IMHO.
User interfaces are not going away.
Yes, humans will use computers alongside agents. When working with agents, we will need the ability to audit what agents do in a human-readable way.
Yes, some interfaces (ex: dashboards, in-chat UI for ephemeral tasks, etc) will be generated. Most important interfaces will be intentionally designed. We will want both deterministic and non-deterministic interfaces in different situations. We're thinking a lot about the best ways to help designers shape non-deterministic systems at Figma.
Yes, new input modalities like voice, VR/AR and BCI will change the way we use computers and introduce lots of new interaction paradigms. This is all very exciting :)
Overall, the surface area of design will continue to grow much broader. Design is the new code and the key differentiator for software moving forward. Expect more and more energy to go into design / craft in the years ahead... it will be how you win or lose.
Eric is right about plenty of things, but the market will prove him wrong on this one...
This is the kind of "AI Magic" I love. Not through delegation and instruction, but through direct interaction with intelligent and dynamic materials.
We have found a local optimum for chat interface; what more do we need than a text box for chat, after all
For many other tasks, shopping, browsing, and complex sensemaking work-rich visual, multimodal interfaces are necessary, and we have not yet found the design pattern for how to integrate them with chat.
Solaris is super cool. It starts to explore what I think of as interface physics.
Working directly with pixels rather than code seems to make it easier to preserve the physics of an interface—how things move, respond, transform, and relate spatially.
Unlike the physical world, where the laws of physics are given, the physics of interfaces can be invented and much more diverse.
Very curious to see how video generation models handle increasingly complex interactions, both within a scene and across scenes.
Today, we're sharing new research on Solaris, our first Interface World Model.
Solaris is a new kind of operating system that generates interactive interfaces frame by frame, in real time, with no code. We find that Solaris outperforms frontier LLMs when generating new interfaces, across structural similarity and information retention. Read more and request early access at the link below.
Haijun Xia retweeted
Interfaces that are generated, not coded.
Excited to finally share Solaris. Video is the universal interface. Eventually, all UI will be chat and gestures in, chat and video out.
Solaris opens new ways to build websites, apps and other interfaces. And most importantly, it gives new ways to train agents in much more dynamic environments.
Interfaces that are generated, not coded.
Solaris is our first step toward our vision of end-to-end neural software, where user interfaces are streamed directly onto your screen by a real-time video model, without any intermediate code or HTML/CSS. Today we're releasing a technical report and opening it up for limited testing by the community.
Amazing work by the team, combining the best of our research in general world models with Runway's HCI spirit.
