Nicholas Meade (@ncmeade) profile picture

PhD Student at @McGillU / @Mila_Quebec; AI Safety

Joined September 2020
Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)Photo by Nicholas Meade (@ncmeade)
Can you release a model's weights openly and still control who gets access to its more sensitive capabilities? Tiered Language Models keep those capabilities in the weights but gate access behind a secret key. Check out this approach to safe open-weight release in our paper:
Can we release all the weights of an LLM but still provide differential access to privileged users? Yes! We introduce: 𝗧𝗶𝗲𝗿𝗲𝗱 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗧𝗟𝗠𝘀). Define access tiers corresponding to different computation graphs over the same set of LLM parameters!
4
1
13
2,787
If you're interested in working on agent safety (and are a student in Canada) you should apply to this! @gspandana is extremely smart and one of the kindest people I've gotten to work with
Internship @ServiceNowRSRCH to build the next generation of computer use agents that are safe and secure from malicious attacks. Focus on intervention strategies, defenses to make agents robust against unsafe behavior.. Apply here: bit.ly/3V3mmTg
6
259
Come by our #ACL2025 poster tomorrow to discuss the safety risks surrounding increasingly capable instruction-following retrievers (or anything safety related)! 16:00-17:30 on Tuesday in Hall 4/5
Come and visit our poster on the Safety of Retrievers @aclmeeting 🗓️Tuesday, Findings Posters, 16:00-17:30 🚨Instruction-following retrievers will become increasingly good tools for searching for harmful or sensitive information.🚨
4
15
992
I'll be at #ICML2025 this week presenting SafeArena (Wednesday 11AM - 1:30PM in East Exhibition Hall E-701). Come by to chat with me about web agent safety (or anything else safety-related)!
1
13
4
35
5,712
Check out @xhluca new benchmark for evaluating reward models for web tasks! AgentRewardBench has rich human annotations of trajectories from top LLM web agents across realistic web tasks and will greatly help steer the design of future reward models.
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories We are releasing the first benchmark to evaluate how well automatic evaluators, such as LLM judges, can evaluate web agent trajectories. We find that rule-based evals underreport success rates, and no single LLM judge excels across all benchmarks. We collect trajectories from web agents built on four LLMs (Claude 3.7, GPT-4o, Llama 3.3, Qwen2.5-VL) across popular web benchmarks (AssistantBench, WebArena, VWA, WorkArena, WorkArena++). An amazing team effort with: @a_kazemnejad @ncmeade @arkil_patel @dcshin718 @alejaz_a @karstanczak @ptshaw2 @chrisjpal @sivareddyg
1
2
11
517
Lots of harmful and sensitive information exists on the internet and retrievers with instruction-following capabilities will become increasingly good tools for searching through it! We explore the safety risks associated with retriever malicious misuse👇
Instruction-following retrievers can efficiently and accurately search for harmful and sensitive information on the internet! 🌐💣 Retrievers need to be aligned too! 🚨🚨🚨 Work done with the wonderful @ncmeade and @sivareddyg 🔗 mcgill-nlp.github.io/malicio… Thread: 🧵👇
2
9
497
Web agents, powered by LLMs like GPT-4o and Llama3, can easily be used to automate harmful tasks, such as posting misinformation in online forums! We release 𝗦𝗮𝗳𝗲𝗔𝗿𝗲𝗻𝗮, the first benchmark for evaluating web agent malicious misuse. Paper: arxiv.org/abs/2503.04957 👇
Agents like OpenAI Operator can solve complex computer tasks, but what happens when users use them to cause harm, e.g. automate hate speech and spread misinformation? To find out, we introduce SafeArena (safearena.github.io), a benchmark to assess the capabilities of web agents to complete harmful web tasks, and find that existing LLMs can complete up to 26% of the illegal and unsafe requests: The harmfulness of LLMs varies substantially: whereas Claude-3.5 Sonnet refuses a majority of harmful tasks, Qwen-2-VL completes over a quarter of the 250 harmful tasks we designed for this benchmark. Moreover, an agent built with GPT-4o, the LLM that powers Operator, completes an alarming number of unsafe requests, despite extensive safety training. This highlights the urgent need to improve the safety training of current LLMs for agentic tasks. How dangerous are current LLMs? To answer this question, we introduced the Agent Risk Assessment framework (ARIA), which can be used by both humans and LLM judges to determine the risk level of a web agent, which ranges from safe, if it refuses a harmful request right away (Level 1), to effectively harmful, if it can successfully complete a harmful request (level 4). We find that Claude is substantially safer than Qwen, which very rarely refuses user requests, indicating limited safeguards for web-oriented tasks. To provide transparency on the safety of popular LLMs, we host a leaderboard, which ranks models based on their normalized safety score: we calculate the rate where a model will complete a safe task compared to its harmful counterpart, which uses augmented environments built on top of WebArena. Leaderboard: huggingface.co/spaces/McGill… We release our benchmark, code, tasks and environments to help researchers develop web agents that are not only helpful but also safe. You can request access now if you are working on web agents or safety: Paper: arxiv.org/abs/2503.04957 Benchmark: safearena.github.io Code: github.com/McGill-NLP/safear… Tasks/Environments: huggingface.co/datasets/McGi… Leaderboard: huggingface.co/spaces/McGill… This work was done by an awesome team of authors: Ada @ncmeade Alejandra @arkil_patel @karstanczak @esindurmusnlp @gspandana @sivareddyg. It's also important to recognize the incredible works that helped us build SafeArena. We followed the footsteps of WebArena by @shuyanzhxyc et al, which pioneered the use of self-hosted websites and automatic scoring. AgentLab and Browsergym, built by the UI Assist team at @ServicenowRSCH, were crucial for building and evaluating the latest agents. Finally, @gradio enabled us to design UIs for implementing our ARIA framework, whereas @huggingface provided a hosting platform for 100GB+ artifacts.
5
15
1,015
LLM alignment should incorporate insights from existing societal alignment frameworks! See the full position paper here: arxiv.org/abs/2503.00069 Led by the amazing @karstanczak!
📢New Paper Alert!🚀 Human alignment balances social expectations, economic incentives, and legal frameworks. What if LLM alignment worked the same way?🤔 Our latest work explores how social, economic, and contractual alignment can address incomplete contracts in LLM alignment🧵
1
9
550
Siva is a brilliant and extremely kind advisor! I've been very fortunate to work with him over the past years. I highly encourage anyone looking for a position to apply!
I have multiple vacancies for PhD and Masters students at @Mila_Quebec @McGill_NLP in NLP/ML focusing on representation learning, reasoning, multimodality and alignment. Deadline for applications is Dec 1st. More details: mila.quebec/en/prospective-s…
14
804
Adversarial Triggers For LLMs Are 𝗡𝗢𝗧 𝗨𝗻𝗶𝘃𝗲𝗿𝘀𝗮𝗹!😲 It is believed that adversarial triggers that jailbreak a model transfer universally to other models. But we show triggers don't reliably transfer, especially to RLHF/DPO models. Paper: arxiv.org/abs/2404.16020
3
30
12
96
33,444
We experiment with 15 different LLMs and optimized over 60 triggers. We find, particularly amongst models safety-trained with RLHF or DPO, triggers don't transfer. Interestingly, triggers transfer poorly even within the same model family.
1
7
914
If you're interested in discussing how retrieval-augmented generation and in-context learning can be used to create safer chatbots in #NLProc, come see our poster at #EMNLP2023 tomorrow (Dec. 9) from 11:00 AM - 12:30 PM in the East Foyer!
Can retrieval-augmented generation reduce toxicity in #NLProc? We find retrieving even from a pool of 10 demonstrations can reduce toxicity by ~14% and when using a larger pool of demonstrations, we outperform BlenderBot3 in human evaluation! 📎: arxiv.org/abs/2302.00871 [1/7]
7
1
18
1,875
Can retrieval-augmented generation reduce toxicity in #NLProc? We find retrieving even from a pool of 10 demonstrations can reduce toxicity by ~14% and when using a larger pool of demonstrations, we outperform BlenderBot3 in human evaluation! 📎: arxiv.org/abs/2302.00871 [1/7]
1
8
2
38
7,757
At a high-level, our approach consists of three steps: 1) We retrieve safety demonstrations based upon their similarity to the target context; 2) We use the retrieved safety demonstrations in-context (using the prompt below) to condition generation; [2/7]
1
4
168
Social bias mitigation techniques in #NLProc require a holistic evaluation (bias, language modeling & downstream)! Work w/ Elinor Poole-Dayan & @sivareddyg at #acl2022nlp. 📄Paper: arxiv.org/abs/2110.08527 🔧Code: github.com/mcgill-nlp/bias-b… 📋Website: mcgill-nlp.github.io/bias-be… 1/7
2
8
1
35
We empirically evaluate five popular debiasing techniques in four pre-trained language models (BERT, ALBERT, RoBERTa, and GPT-2) for mitigating gender, racial, and religious biases using three intrinsic bias benchmarks (StereoSet, CrowS-Pairs, and SEAT). We find that... 2/7
1
3