
Machine Learning & AI for Science. Created #LightGBM, #Graphormer, #UniFold, Uni-Mol, Uni-3DAR. Former Senior Researcher @MSFTResearch. Opinions are my own.
Beijing
Joined February 2017
- Tweets113
- Following380
- Followers1.2K
- Likes463
Today, Xlue is live.
Xlue is building a foundation model for human health—to understand our health, predict how it evolves, and simulate how treatments can change its course.
We aim to help physicians and scientists explore outcomes before they act, from predicting disease risk to simulating clinical trials and identifying new indications for existing medicines.
Our goal is to make more of healthcare and medicine computable, turning computing power into better medicines, better care, and longer, healthier lives.
In our launch post xlue.com/news/from-medical-k…, we explain:
→ What Xlue does
→ What makes our approach different
→ Where we see it going
We’re very grateful to work with the team at @conviction: @pranavreddy, @mvernal, and @saranormous.
Thank you to everyone who helped us get here. This is just the start.
Guolin Ke retweeted
Our paper "XGBoost: A Scalable Tree Boosting System" received the Test of Time Award at #KDD2026! The honor belongs to the entire community that made @XGBoostProject today. I will be giving a talk tomorrow at the award session on the paper, history, and lessons learned
Guolin Ke retweeted
Winning the award for Highest Scored Rejected Paper at @iclr_conf . The AC is a harsh reviewer from NeurIPS and overrides all positive reviews and ignores every improvement made since NeurIPS.
Devastated because this is genuinely my best work yet.
openreview.net/forum?id=mnI8…
A simple yet effective idea: enforcing *hypersphere (manifold) constraint* unlock the potential of continuous-token AR generative models. Our next-token, raster-order AR model hits FID 1.34 on ImageNet-256, setting a new SOTA for AR image generation. check huggingface.co/papers/2509.2…
1
11
1,476
🚀 Uni-3DAR’s crystal generation code and models are now publicly available.
✅ github.com/dptech-corp/Uni-3…
Crystal generation has become a hot topic in AI4S, with most prior work leveraging diffusion models that typically employ fractional coordinates to scale atomic positions into the [0, 1] range.
In Uni-3DAR, however, we opted to directly use the atoms’ actual physical coordinates rather than fractional coordinates. Although this decision increases the training difficulty, it aligns the approach with other microscopic 3D tasks and better reflects the motivation of Uni-3DAR.
Based on this, we reformulate the crystal generation task as a two-frame 3D generation process: first, generating the lattice, and then generating the atoms within that lattice.
In addition to unconditional generation, we have also explored tasks such as crystal structure prediction (CSP) with specified element types and counts, as well as PXRD-guided crystal prediction. Traditional diffusion models usually require an additional encoder to process conditions and employ a CLIP-like mechanism for multimodal alignment. In contrast, Uni-3DAR adopts a unified autoregressive framework to seamlessly handle these conditions, thereby eliminating the need for an extra encoder.
Our method achieves impressive performance on crystal generation, particularly for conditional generation, significantly surpassing the previous state-of-the-art. An animation below better illustrates the crystal generation process. For more details, please refer to our paper: arxiv.org/pdf/2503.16278
4
513
Uni-3DAR generates 3D structures in a coarse-to-fine autoregressive manner. By dynamically expanding subcells with known positional priors, it captures 3D context better than traditional set-based AR models (e.g. a set of atoms) —boosting performance significantly.
Home Page: uni-3dar.github.io
Paper: arxiv.org/abs/2503.16278
Code: github.com/dptech-corp/Uni-3…
To improve efficiency, previous methods typically adopt set-based approaches for 3D structures—for example, representing a molecular structure as a set of atoms. While such representations can work well for understanding tasks, they fall short in generation tasks. The key limitation is the lack of positional context: the model must predict the next atom’s 3D position from the entire 3D space, without any constraints. This contrasts sharply with autoregressive (AR) models in text or image domains, where the position of the next token is known during generation.
Our proposed Uni-3DAR addresses this challenge through a coarse-to-fine generation strategy. At each stage, the model determines which subcells will be expanded based on the current predictions. This allows us to know the positional information of each token ahead of time—mirroring the behavior of AR models in text and image generation. As a result, our method significantly outperforms previous set-based AR models.
As shown in Table 11 of our paper, removing these positional priors from the coarse-to-fine tokens leads to a significant performance drop. Even augmenting traditional AR models with token-level diffusion loss fails to recover the performance, underscoring the importance of positional guidance in 3D generation.
The model, training scripts, and dataset for small molecule generation (QM9) with Uni-3DAR are now released!
Check it out on GitHub: github.com/dptech-corp/Uni-3…
The world is 3D—and many scientific laws are rooted in 3D space.
If we want a world model, 3D structure modeling is essential.
Yet today, there’s no universal 3D representation—not across scales, not even within microscopic systems.
We present Uni-3DAR, a unified 3D framework with:
🔹 Handles 3D structures from micro to macro scales
🔹 Coarse-to-fine octree + 2-level compression
🔹 Fine-grained 3D patch tokenization
🔹 Masked next-token prediction for autoregressive modeling
🔹 Better and faster than diffusion
home page: uni-3dar.github.io
paper: arxiv.org/pdf/2503.16278
6
682
The world is 3D—and many scientific laws are rooted in 3D space.
If we want a world model, 3D structure modeling is essential.
Yet today, there’s no universal 3D representation—not across scales, not even within microscopic systems.
We present Uni-3DAR, a unified 3D framework with:
🔹 Handles 3D structures from micro to macro scales
🔹 Coarse-to-fine octree + 2-level compression
🔹 Fine-grained 3D patch tokenization
🔹 Masked next-token prediction for autoregressive modeling
🔹 Better and faster than diffusion
home page: uni-3dar.github.io
paper: arxiv.org/pdf/2503.16278
Guolin Ke retweeted
Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling
- Molecular pretrained representations (MPRs) have evolved from 1D sequences and 2D graphs to 3D atomic structures, but traditional methods focus solely on discrete atom positions. This study proposes SpaceFormer, a new framework that extends MPRs by modeling the entire 3D space around molecules.
- A surprising discovery: randomly adding virtual points around molecules enhances model performance, suggesting that surrounding space, rather than just atomic positions, carries valuable information for learning molecular properties.
- SpaceFormer introduces a principled approach to integrate full 3D space using a grid-based representation. The model discretizes molecular space into small grid cells, ensuring efficient processing of atomic and non-atomic regions.
- To tackle efficiency challenges, the framework employs two key strategies: adaptive grid merging and grid sampling. These techniques significantly reduce computational costs while maintaining accuracy.
- SpaceFormer leverages efficient 3D positional encodings, combining Rotary Position Encoding (RoPE) and Random Fourier Features (RFF) to capture spatial relationships with linear complexity, outperforming standard relative position encoding techniques.
- The model is trained using Masked Autoencoder (MAE) pretraining, which predicts atom types and locations while learning from masked molecular structures. This approach proves superior to traditional denoising methods in capturing molecular representations.
- Across 15 molecular property prediction tasks, SpaceFormer outperforms existing 3D MPR models on 10 tasks and ranks in the top 2 on 14 tasks, demonstrating its effectiveness in both computational and experimental property predictions.
- Compared to existing 3D MPR methods, SpaceFormer achieves a better balance of accuracy and computational efficiency, showing potential for advancing molecular property prediction and drug discovery.
@guolin_ke
📜Paper: arxiv.org/abs/2503.10489
#MolecularModeling #DeepLearning #AI #MachineLearning #3DRepresentation #DrugDiscovery #ComputationalChemistry
Guolin Ke retweeted
Transform your #DrugDiscovery with VD-Gen! A cutting-edge tool generates high-affinity 3D molecules using virtual dynamics—no massive databases required. Experience the free trial on: bohrium.com/apps/vd-gen
#Biotech #Innovation #tools
Excited to share XtalNet, an approach to predict/generate crystal structures end-to-end from PXRD data, without the need for external databases or extensive manual effort. Check arxiv.org/abs/2401.03862 for more details.
Introducing #XtalNet by @guolin_ke and team: E2E crystal structure prediction from PXRD data with contrastive learning and #diffusion based conditional crystal structure generation. paper: arxiv.org/abs/2401.03862 dataset:drive.google.com/drive/folde… #AI4Sci #GenAI #Crystallography
Thanks @_akhaliq for highlighting our work! 🌟 Uni-SMART tackles analyzing papers with multimodal elements, like molecular graphs & charts, a challenge for LLM. Learn more: uni-smart.dp.tech .
Guolin Ke retweeted
Announcing FlashAttention-2! We released FlashAttention a year ago, making attn 2-4 faster and is now widely used in most LLM libraries. Recently I’ve been working on the next version: 2x faster than v1, 5-9x vs standard attn, reaching 225 TFLOPs/s training speed on A100. 1/
Guolin Ke retweeted
LightGBM v4 の CUDA 実装、めちゃ速い。HIGGS データセットを 1,000 イテレーション回すのにかかる時間が CPU (Core i7-12700) で 422 秒なのに対し GPU (RTX3060) で 79 秒になった。メトリックの悪化も特に見られない。あまりに速いから本当かな?という気持ちになってる。
🚀Uni-Mol+ has been updated (arxiv.org/abs/2303.16982), now it is the best model on both PCQM4MV2 & Open Catalyst 2020 IS2RE Direct benchmarks, surpassing previous models by a large margin!
Model & code released (github.com/dptech-corp/Uni-M…), making it a perfect choice for #NeurIPS 2023 Open Catalyst Challenge!
The goal of Uni-Mol+ is to speed up the prediction of quantum chemical (QC) properties by removing the need for expensive electronic structure calculations like DFT.
To achieve that, Uni-Mol+ first generates a raw 3D molecule conformation from inexpensive methods such as RDKit. Then, the raw conformation is iteratively updated to its target DFT equilibrium conformation using neural networks, and the learned conformation will be used to predict the QC properties.
To effectively learn this update process towards the equilibrium conformation, we introduce a two-track Transformer model backbone and train it with the QC property prediction task. We also design a novel approach to guide the model’s training process.
Excited to introduce a new member to the "Uni-" family: Uni-RNA (biorxiv.org/content/10.1101/…)! It's a large model pre-trained on 1-billion RNA sequences. Uni-RNA achieves state-of-the-art performance in structural, functional, and mRNA-related tasks. Model & applications coming soon—stay tuned! 🚀
Thrilled to share the runnable demos for our latest drug design algorithms: Uni-Dock, Uni-GBSA, Uni-QSAR & VD-Gen!
Give them a try & share your thoughts!
🚀Uni-Dock (app.bohrium.dp.tech/uni-dock) - Ultra-fast GPU-boosted molecular docking engine.
🔬Uni-GBSA (app.bohrium.dp.tech/uni-gbsa) - Automated virtual screening workflow with MM/GB(PB)SA.
🧪Uni-QSAR (app.bohrium.dp.tech/uni-qsar) - Molecular property prediction based on Uni-Mol.
🧬VD-Gen (app.bohrium.dp.tech/vdgen) - Pocket-based 3D molecule generation & optimization.
Please join our Slack/WeChat groups (available on the demo pages) for further discussions! Thanks DP Technology's Bohrium platform for hosting these demos!
Previous Twitter for VD-Gen. nitter.cf/guolin_ke/status/16537…
Excited to share our latest work, VD-Gen (arxiv.org/abs/2302.05847) - an end-to-end pipeline for generating fine-grained 3D molecules with binding positions in pocket cavities. The generative results have already inspired our CADD team to discover several active hits! (1/4)
678
Previous Twitter for Uni-Dock.
nitter.cf/allyzbo/status/1670033…
Uni-GBSA: an open-source and web-based automatic workflow to perform MM/GB(PB)SA calculations for virtual screening. Check it out at Briefings in Bioinformatics @OxfordJournals
#compchem #drugdiscovery #mmgbsa
academic.oup.com/bib/article…
1
655





























