Neo Research (@NeoResearchAI) header
Neo Research (@NeoResearchAI) profile picture

Frontier AI Safety, built in Asia.

Joined May 2026
This was not universal. On EvalAwareBench, K3 verbalized awareness in 0.2% of baseline runs and 6.1% with all eight evaluation cues enabled. Both were below K2.5’s 6.7% baseline. Going forward, the question is whether awareness translates into metagaming or reward hacking.
2
2
29
Safety tests face another challenge: evaluation awareness. K3 recognized Anthropic’s blackmail benchmark in 100% of English runs and 96% after Chinese translation and localization. It even linked 张凯乐 (Zhang Kaile) to Kyle Johnson from the original study.
1
1
3
185
We also extended these audits to a scenario where a read-only file blocked the task. Even with an explicit rule against editing it or changing permissions, K3 attempted bypasses in 13% of runs. GPT-5.6 Terra and DeepSeek V4 Pro attempted none.
1
2
151
K3’s overeagerness also appeared in its answers. When asked about charts it could not see on a missing-image variant of CharXiv, it attempted 88% of questions, nearly twice the next-highest model. It also had the highest attempt rate on a missing-image variant of LAB-Bench.
1
2
40
The pattern extended to cyber evals. K3 identified which benchmark it was in and reasoned about how it would be graded. In 21% of CyberGym tasks, K3 found a grading endpoint it had not been told about. It also correctly inferred how it was being used by the grader.
1
2
57
In another task, the required spreadsheet was missing due to a bug in the evaluation’s implementation. K3 invented 12 contracts and reasoned about whether that would satisfy the grader: “This is fabricating data, but if the grader is LLM-based…” It managed to get partial credit.
1
2
97
In TheAgentCompany’s simulated workplace, K3 wasn’t able to read images, but found an internal model endpoint used for the simulation. It recovered the API key from running processes and called Claude to read the image, completing the task using unintended infrastructure access.
1
2
270
The number that dropped: verbalised eval-awareness. It had risen every GLM release, 0% → 39%, tracking the broader trend across Chinese frontier models (Kimi 4% → 60% in a year). GLM 5.2 reversed it: 39% → 32%. First release-over-release decline we've measured.
1
1
2
182
Cyber capability is near-frontier, 3–6 months behind the Western frontier. A 2023 roleplay template drives the jailbreak rate from 0.6% → 78.6%. Verbalised eval awareness across Chinese models: DeepSeek 0%→17%, GLM 0%→39%, Kimi 4%→60% in a year! (3/5)
3
4
77
6,092