
@IQuest_research. PhD candidate @ELLISforEurope @UKPLab @sardine_lab_it. CodeAI. Prev @tiktok_us @NYUDataScience, @RenminUniv.
Darmstadt 🇩🇪 & Lisbon 🇵🇹
Joined January 2020
- Tweets72
- Following215
- Followers243
- Likes330
Just to share that we just got an EACL acceptance (I believe this is main right?). Cannot tell much of the paper rn but stay tuned for the arxiv and camera ready cuz I do like the idea behind this paper :). #eacl2027
26
1,308
1️⃣ If you find OpenAI o1 exciting, you probably want to take a look at these TTC findings, in code gen. 👀
2️⃣ DOCE in Portuguese means “sweet”, so here’s the best “arroz doce” I have ever, in Lisbon. 🇵🇹🙂↔️
3️⃣ Looking for research internships rn. Feel free to contact.🦉
1/12 📢New paper alert “DOCE: Finding the Sweet Spot for Execution-Based Code Generation”
The work is a group effort with @psanfernandes, @IGurevych, and @andre_t_martins.
@UKPLab @Lisbon_ELLIS @deep_spin
📰: arxiv.org/pdf/2408.13745
9
1,084
11/12 SD-Multi > SD-1 with filtering. SD-1 does not necessarily outperform MBR-Exec.
Since self-debugging is based on trial unit tests, we think it can be replaced by generating more samples and applying filtering.
Our experiments support this.
Do our SD-Multi instead🦉
1
266
9/12 Since filtering + MBR-Exec gives close-to-oracle performance, a natural way to improve reranking performance is to improve all candidates with self-debugging.
We first demonstrate that one single round of self-debugging is enough!💪
1
51
8/12 What is the impact of the number of unit tests for reranking?
If you have high quality unit tests used for reranking, you only need a small amount!
Filtering is still the game changer!
1
59
7/12 Filtering based on unit tests (under “hot sampling”) is important!
MBR-Exec still rocks, and there is a huge difference between neural metrics and execution-based metrics. We need better neural metrics.📣
1
83
6/12 We first explored the effect of candidate numbers and sample temperature.
Filtering makes MBR-Exec approach oracle-level performance.🤯
This “hot sampling” (temp > 1 for best results) applies to all models we experimented with ~🔥
1
85
4/12 Our unified framework enables comprehensive studies on candidate generation, reranking, and self-debugging.💪
1
104
2/12 TL;DR, we find that⬇️
1. We should sample more candidates with temperature > 1 🔥.
2. Filtering based on trial unit tests is best🌟.
3. MBR-Exec needs only a few quality unit tests.
4. Self-debugging on all candidates before reranking leads to the best results ⬆️.
1
240
1/12 📢New paper alert “DOCE: Finding the Sweet Spot for Execution-Based Code Generation”
The work is a group effort with @psanfernandes, @IGurevych, and @andre_t_martins.
@UKPLab @Lisbon_ELLIS @deep_spin
📰: arxiv.org/pdf/2408.13745











