Ding Junhua

What if you can improve LLMs using direct customer feedback? 🤔 Most alignment methods like RLHF or DPO require multiple outputs from the same prompt to improve the model by learning the preferred one. 👀 In real-world use cases, you oftentimes only have the option to collect one-dimensional feedback, e.g., whether the response was good or bad. This made using techniques like DPO and RLHF on real-world data only hard. 💬 KTO tries to simplify this by using binary feedback on individual outputs. 👍🏻👎🏻 This is less complex and can be directly used on real-world data. 🌍 How to use KTO: 1️⃣ Collect binary feedback data 2️⃣ Create a balanced (50/50) dataset out of triplets (prompt, response, feedback) 3️⃣ Apply KTO to a fine-tuned LLM using the dataset (2️⃣) 4️⃣ Evaluate outputs from the KTO model vs the old model using LLM as a judge or humans. Insights: 👩‍🔬 @ContextualAI ran 56 experiments using DPO, KTO, and PPO (RLHF) 🚀 KTO matches the performance of DPO & PPO 🦙 KTO showed better performance on bigger models (llama 1) 💡 Simplifies data requirements compared to DPO & RLHF ⚖️ Need a balanced binarized dataset (50% good 50% bad examples) Report: github.com/ContextualAI/HALO… Github: github.com/ContextualAI/HALO… KTO will soon be available in @huggingface TRL. 🤗 Huge Kudos to @douwekiela and the team from @ContextualAI for developing this new alignment method. 🚀