The first hardware benchmark built in partnership the world’s most advanced hardware companies, created by @arenaphysica 💡
New York, USA
Joined September 2026
- Tweets27
- Following1
- Followers108
- Likes25
HW-Bench retweeted
What the antenna results show:
1️⃣ 11 of 15 models never passed every critical requirement on all five runs of any task. GPT-6 Astra did it on four of six. Hardware can only ‘compile once,’ so a model that’s only right some of the time can cost a full fab cycle.
2️⃣ No single model is best. Six different models had the top accuracy score on at least one of the six tasks, and sending each task to its most accurate model scored 0.875, vs. 0.805 for GPT-6 Astra on every task, at 73% less cost.
3️⃣ Driving simulation and EDA tools is the hardest part. Only 13% of runs on the HFSS and KiCad tasks passed every critical criterion, vs. 41% on the text and code tasks.
4️⃣ Open-weight models trail. GLM-5.3 and Kimi K3 ranked 12th and 15th by accuracy, yet cost more than twice as much as GPT-6.1 Sol, the second most accurate model.
Today, we're releasing @hardwarebench, the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
Measurement has always been core to our work at @arenaphysica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks.
HW-Bench evaluates frontier model performance on production hardware data and captures the full hardware context across tools and file formats. Each task is graded against rubrics and golden answers reviewed by subject matter experts at our partner companies, who build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond (more coming soon).
We'll be building towards jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. To start, see how 15 frontier models performed at designing a 2.45 GHz antenna at hw-bench.com.
HW-Bench retweeted
love this. we did the machining side: nitter.cf/MerkleAnders/status/21…
Introducing RealShop-CAD, a benchmark for measuring how well AI turns real production drawings into parts a machine shop can actually make.
Built from anonymized production drawings from MES customers, RealShop-CAD includes 228 drawing-to-STEP tasks, graded to 0.1 mm against each part's production 3D model.
HW-Bench retweeted
LLMs still struggle with real engineering tools.
On HFSS/KiCad tasks, only 13% of runs passed every critical check vs 41% on text tasks. On PCB layout, just 3 of 75 runs passed, and most set the wrong board stackup.
HW-Bench retweeted
Arena Physica launched HW-Bench, the first hardware engineering benchmark built with top hardware companies.
It grades frontier models on production hardware data across real tools and file formats.
Software benchmarks are saturated. Hardware exposes models
Today, we're releasing @hardwarebench, the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
Measurement has always been core to our work at @arenaphysica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks.
HW-Bench evaluates frontier model performance on production hardware data and captures the full hardware context across tools and file formats. Each task is graded against rubrics and golden answers reviewed by subject matter experts at our partner companies, who build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond (more coming soon).
We'll be building towards jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. To start, see how 15 frontier models performed at designing a 2.45 GHz antenna at hw-bench.com.
HW-Bench retweeted
Finally someone working on non-RTL benchmarks! now we need HIL testing
Today, we're releasing @hardwarebench, the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
Measurement has always been core to our work at @arenaphysica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks.
HW-Bench evaluates frontier model performance on production hardware data and captures the full hardware context across tools and file formats. Each task is graded against rubrics and golden answers reviewed by subject matter experts at our partner companies, who build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond (more coming soon).
We'll be building towards jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. To start, see how 15 frontier models performed at designing a 2.45 GHz antenna at hw-bench.com.
HW-Bench retweeted
Arena Physica just launched HW-Bench, a benchmark that tests AI models on real hardware engineering. Here's what you need to know.
HW-Bench is billed as the first hardware engineering benchmark built with the world's most advanced hardware companies. It uses production hardware data and captures the full hardware context across tools and file formats. Every task is graded against rubrics and golden answers reviewed by subject matter experts at partner companies, with more partners coming. Arena Physica says measurement has always been core to its work, and the first version of its Atlas product shipped with an internal evaluation framework.
The launch results cover one job, designing a 2.45 GHz antenna, split into six tasks. The tests ran in HFSS and KiCad. Fifteen frontier models were scored on accuracy, cost and latency. The scoring is strict. Pass^k is the chance that k runs of the same task all pass every critical criterion, so failing one critical criterion fails the run.
The team says the next goal is jobs that span disciplines, like designing, manufacturing and testing a 140 GHz, 512-element D-band phased array. They are also asking engineers, companies with designs or data, and researchers to contribute. Rankings may shift as more jobs are added.
Top of the leaderboard:
- 1st: OpenAI GPT-6 Astra, 67% pass rate, 0.81 accuracy, $50.49 per job, 1h 28m
- 2nd: GPT-6.1 Sol, 57% pass1, 42% pass3, 33% pass5, 0.78 accuracy, $8.42, 1h 45m
- 3rd: GPT-6 Sol, 37% pass1, 0.71 accuracy, $8.07, 53m
- 4th: Anthropic Claude Opus 5, 33% pass1, 3% pass3, 0.71 accuracy, $28.87
- Claude Fable 5.1: 23% pass1, $33.84. Claude Sonnet 5.5: 20% pass1, $5.75
- GPT-6 Luna: 30% pass1 at just $0.55 per job
One contributor says no single model wins. Routing each task to its best model scored 0.875 accuracy versus 0.805 for the top single model, at 73% lower cost.
Other voices say teams should build benchmarks on their own data, and that benchmark results shouldn't be trusted blindly. HW-Bench launched on Nov 5, 2026, at hw-bench.com.
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
hardware has been the blind spot in most benchmarks, good to see someone go after it properly
HW-Bench retweeted
Cool new LLM model benchmark just dropped!
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
One of the most fun projects I've worked on is live: HW-Bench, 15 models on a real antenna design job in HFSS and KiCad.
No single model wins: routing each task to its best model beat the top model alone, 0.875 vs 0.805 accuracy at 73% lower cost.
hw-bench.com
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
RT @_i_am_arya: Our Why’s behind @hardwarebench at @arenaphysica:
- Verifiable, by design…get your judges out of here
- Multi-discipline,…
This quoted post is unavailable.
HW-Bench retweeted
There are no good benchmarks that measure how good AI really is at hardware.
A useful benchmark needs to be grounded in real workflows, for real products, that exhibit real complexity. Today, we're excited to introduce HW-bench, the result of partnering with some of the most advanced hardware teams to make a useful benchmark. Early results are quite interesting:
1. An “index” fund of models is best. Across the 15 frontier closed & open models we benchmarked, no single model excelled at a job. Routing across models always delivered the best results. Jobs are baskets of tasks that map to the work actual engineers do.
2. Mean accuracy is a misleading metric. We ran each model 5 times, and only 4 models completed the task every time. Working “sometimes”, well, doesn’t cut it for hardware
3. Models are great at text in/text out, and simple tool use. But, testing tool use for simulation and EDA really separates the field. Just 13% of runs on Ansys HFSS (EM simulation) and KiCad (EDA) tasks passed every critical criterion, vs. 41% for text & code tasks
4. The gap to open source is real – open models are not just significantly worse, but they are more expensive
We'll be adding more jobs as we go. If there's a workflow you'd like to benchmark, reach out
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
HW-Bench is now live and I’m so proud of what our team put together @arenaphysica
Checkout what we’ve been up to
@hardwarebench: hw-bench.com
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
Good idea!
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
Interested in building HW-Bench with us? We're hiring for our Evals team at arenaphysica.com/careers!
HW-Bench retweeted
Today, we're releasing @hardwarebench, the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
Measurement has always been core to our work at @arenaphysica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks.
HW-Bench evaluates frontier model performance on production hardware data and captures the full hardware context across tools and file formats. Each task is graded against rubrics and golden answers reviewed by subject matter experts at our partner companies, who build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond (more coming soon).
We'll be building towards jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. To start, see how 15 frontier models performed at designing a 2.45 GHz antenna at hw-bench.com.
HW-Bench retweeted
Measurement has always been core to our work at @arenaphysica and we're excited to release @hardwarebench today!
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon).
Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com!
Created by the team @arenaphysica ⚡
HW-Bench retweeted
Excited to share @hardwarebench with the world!
Whether you’re an engineer who wants to help shape the framework, a company with designs or data to contribute, a researcher who wants to build on HW-Bench or someone curious about how frontier models perform on real hardware work, sign up for updates at hw-bench.com or reach out to learn more.
Today, we're releasing @hardwarebench, the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies.
Measurement has always been core to our work at @arenaphysica: the very first version of Atlas shipped with our internal framework for evaluating frontier models on real hardware engineering tasks.
HW-Bench evaluates frontier model performance on production hardware data and captures the full hardware context across tools and file formats. Each task is graded against rubrics and golden answers reviewed by subject matter experts at our partner companies, who build some of the most advanced hardware in the world, from the chips behind modern supercomputing to the autonomous systems and vessels redefining defense at sea and beyond (more coming soon).
We'll be building towards jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. To start, see how 15 frontier models performed at designing a 2.45 GHz antenna at hw-bench.com.