AI models on real semicha exam questions from Israel's Chief Rabbinate, in Hebrew. Provisional results; reviewers welcome: [email protected]
Joined October 2026
- Tweets64
- Following24
- Followers916
- Likes56
Pinned Tweet
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet.
The best scored 80 out of 100. Many scored under 11. ๐งต
Quick guess: which semicha paper did AI find hardest?
Answer tomorrow, with the chart.
18%Chuppah v'Kiddushin
35%Niddah
30%Mikvaot
18%Tefillah u'Brachot
40 votes โข 21 hoursGrok is next on HalachaBench. The top score so far is 80 out of 100 (Claude Fable 5.1).
Where will Grok land?
8%New #1 (above 80)
41%65 to 80
28%Under 65
24%Just show me
143 votes โข 7 hours HalachaBench retweeted
AI takes rabbinical ordination examsโฆ ๐
Thank you to @SefariaProject: before the grading sheets were sealed, an AI checked their cited sources against Sefaria's library, about 2,000 lookups. Sources Sefaria doesn't carry are marked unverified.
Ask an AI the same semicha question three times and you can get three different answers.
Keep only each model's weakest try on every question and scores drop 7 to 11 points. The order doesn't change.
You'd want the same answer every time you ask.
ALT Chart: score out of 100 for each AI, average of its three tries vs. its worst try on each question. Claude Fable 5.1 79.7 vs 70.4; GPT-6 Astra 73.3 vs 66.2; GPT-6.1 Sol 72.8 vs 63.9; Claude Opus 5.5 69.1 vs 60.0; Gemini 3.8 Flash 64.4 vs 54.3; GPT-6.1 Terra 34.6 vs 25.6; GPT-6 Luna 32.4 vs 21.8; Claude Sonnet 5.5 28.7 vs 18.3. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.
ืืืืจื ืืช HalachaBench: ื ืชื ื ืืืืืืื ืฉื ืืื ื ืืืืืืชืืช (ืฉื ืืืฆืจื ChatGPT, Claude, Gemini ืืขืื) ืฉืืืืช ืืืืชืืืช ืืืืื ื ืืืกืืื ืฉื ืืจืื ืืช ืืจืืฉืืช. ืื ืขื ื ืืขืืจืืช, ืืื ืกืคืจืื ืืืื ืืื ืืจื ื.
ืืืืื ืืืื ืืืืชืจ ืงืืื 80 ืืชืื 100. ืจืืื ืงืืืื ืคืืืช ื-11. ๐งต
ืืฉืื ืืืืืืฉ: ืืื ืชืจืืื ืืื ืืืงืืืืื ืืืื. ืืืฉ ืืื ื ืืืฉื ืฉืืื ื ืืืืืืชืืช ืืืืื ืืคืกืืง ืืืื.
ืืชืืฆืืืช ืจืืฉืื ืืืช ืืืื ืืืช: ืืืื ืืื ืืชื ืืช ืืืืืื ืื ืืืืื ืืืจ ืืืง ืืช ืืชืฉืืืืช, ืืฉื ืืื ืื ื ืืื ื ืืขืฆืื. ืืฃ ืจื ืขืืืื ืื ืขืืจ ืขื ืืืืืื ืื ืื ืขื ืืฆืืื ืื.
ืื ืื ืฉืจืืฆื ืืขืืืจ ืขื ืืืืืื ืื ืื ืืืืืง ืชืฉืืืืช: ื ืฉืื ืืืื ืืขืืจืชืื. ืจืืฆืื ืืขืืืจ ืืืจืืืช ืืืืื? ืืชืื ืื ื: [email protected]
ืืชืืฆืืืช ืืืืืืช ืืคืืจืื ืืฉืืื (ืืื ืืืืช, ืขื ืกืืืื ืืขืืจืืช): github.com/halachabench/halaโฆ
