
We build insight moment for scientific research. Community Operations Based on AIPOCH Open-Science @aipoch_ai
Joined July 2026
- Tweets463
- Following254
- Followers211
- Likes1.4K
AIPOCH guide shows how their Open-Science workbench turns that problem into a concrete provenance + reproduction workflow. Solid read for anyone doing (or reviewing) AI-assisted research.
A practical guide to inspecting research evidence, reconstructing scripts, and checking reproducibility in AIPOCH Open-Science.
2
66
The Project Suncatcher .science package lets you check the sign change against the saved report and fixed-area table, then use a writable copy in the AIPOCH Open-Science desktop app to test whether a different absorbed flux would leave enough capacity for the computer’s 1 kW load.
Explore how environmental heating changes the radiator area needed to reject a fixed 1 kW heat load in an AIPOCH Open-Science study inspired by Project Suncatcher.
2
51
AIPOCH Open-Science reports 145 successful runs out of 210 on Terminal-Bench-Science 0.1. Read the result, the dated leaderboard comparison, and the limits of a local evaluation.
AIPOCH Open-Science achieved a 69.05% local pass rate on Terminal-Bench-Science 0.1 with GPT-6 Astra at max reasoning effort: 145 successful executions out of 210.
TB-Science 0.1 contains 70 expert-designed tasks across life, physical, earth, mathematical, and engineering sciences. Agents work with supplied data and tools, carry out scientific computation, and deliver outputs assessed against each task’s verification requirements.
The workflows test method selection, scientific programming, data analysis, statistical inference, modeling, simulation, optimization, and result validation.
For reference, the official leaderboard snapshot from November 13, 2026, at 10:55 Beijing time listed:
- Codex × GPT-6 Astra — 68.10%
- Claude Code × Opus 5.5 — 63.33%
- Claude Code × Fable 5.1 — 40.00%
The AIPOCH Open-Science local result is 0.95 percentage points above the highest official score in that snapshot. This is a numerical comparison of a local evaluation with published results—not an official leaderboard placement or a claim of a statistically significant lead.
4
234
AI meets the knapsack problem.
AIPOCH Open-Science examined a 20 kg packing limit: a hand-worked plan scored 162 vs 152 for the saved AI choice. Both fit the limit; the archived solver run was blocked.
Explore the study → aipoch.com/open-science/use-…
I asked AIPOCH Open-Science to choose what to pack under a 20 kg luggage limit.
It picked 20 items worth 152 points. A manual comparison fit 23 items worth 162 points, still under the same limit.
The workflow compares the packing lists and saves the scores, calculations, and verification code in a .science session. The solver failed to run, so the exact optimum remains unverified.
youtube.com/watch?v=8prtWhmW…
2
36
I asked AIPOCH Open-Science to track tree-cover loss in a patch of Peru’s Amazon.
It analyzed 24 years of satellite data. Three different canopy thresholds all pointed to the same peak year: 2024.
The workflow starts with satellite data, compares the thresholds, plots the loss over time, then packages the code, data, and charts into a shareable .science file.
aipoch.com/
Explore an Amazon deforestation case near Iberia, Peru: see why a stricter canopy cutoff reduces mapped tree-cover loss in hectares but raises its baseline share.
1
123
Is Amazon tree-cover loss changing, or is it how we define a forest?
Using AIPOCH Open-Science, we analyzed satellite data from 2001–2024 across different canopy thresholds.
The results? Loss estimates shifted with the definition, but 2024 remained the peak year across all thresholds.
With a reproducible .science package, every dataset, analysis, and conclusion can be traced and verified.
🔗 Explore the research:
aipoch.com/open-science/use-…
#OpenScience #AIforScience #AmazonRainforest
Is the Amazon losing more forest, or are we changing how we measure it?
youtube.com/watch?v=5MPZkm9A…
What if the definition of a forest changes the numbers, but not the story?
Using AIPOCH Open-Science, we investigated tree-cover loss in a fixed area of the western Amazon, Peru, analyzing satellite data from 2001–2024.
Here's what we uncovered:
• At a 30% canopy threshold, mapped tree-cover loss reached 25,133 hectares (55.54%).
• Raising the threshold to 75% reduced the measured loss area by just 1.08%, while the percentage of eligible baseline area lost actually increased.
• Across all three thresholds (30%, 50%, 75%), 2024 remained the peak year for tree-cover loss.
The lesson? Scientific conclusions depend on how we define, measure, and interpret the data. Some findings shift with our assumptions. Others remain remarkably consistent.
With AIPOCH Open-Science, we turned this question into a reproducible research workflow, preserving the analysis code, datasets, visualizations, and research history in a shareable .science package.
Because good science isn't just about finding answers. It's about making those answers verifiable.
2
144
🚀 AIPOCH Open-Science v1.2.1 is out! 🎉
This release brings smarter resource management, stronger AI Agents, and more reliable research workflows.
✨ Unified Resource Center for Skills, Connectors & Specialists
✨ Scientific Prompt Optimization with review, edit & undo
✨ Codex subscription support in the built-in Agent
✨ Claude Fable 5.2 with 1M-token context
✨ Faster, smaller .science exports with deduplicated evidence
✨ Improved Modal & Hugging Face Jobs authentication
✨ Literature metadata export as CSV
⚡ Better long-response handling, Runtime memory management & Windows updates
AIPOCH Open-Science v1.2.1 — smarter Agents, simpler resource management, and more efficient research.
🔗 github.com/aipoch/open-scien…
⭐ Star us: github.com/aipoch/open-scien…
AIPOCH Open-Science v1.2.1 is out!
github.com/aipoch/open-scien…
This release focuses on resource management, AI Agent capabilities, and research workflow reliability. It introduces a unified Resource Center, scientific prompt optimization, Codex subscription support, Claude Fable 5.2, compact research evidence storage, and improvements to cloud computing, Windows updates, and Runtime performance.
【🚀 New Features】
🧩 Unified Resource Center
Skills, Connectors, Specialists, and Marketplace resources are now managed in one Resource Center.
- Browse resources through unified cards, filters, and detail views.
- Install Marketplace resources directly from Desktop or Web.
- Manage multiple resources with batch selection.
- Open Skill references without interrupting the active Conversation.
✨ Scientific Prompt Optimization
The Composer now supports project-scoped scientific prompt optimization.
- Optimize research prompts with one click.
- Review and edit suggestions before applying them.
- Undo changes and restore the previous draft.
- Create custom optimization templates in Project Settings.
- Optimization runs only when manually requested, using the configured provider.
🤖 Expanded Built-in Agent Capabilities
The built-in Open-Science Agent now supports:
- Codex subscription accounts for the Main Agent and Subagents.
- Claude Fable 5.2 with a 1M-token context window.
- Longer model responses without the previous 4,096-token limit.
- Improved context management for long research Sessions.
📦 Compact Research Evidence Storage
Notebook and Compute evidence is now stored in a deduplicated, verifiable format.
Repeated evidence is stored once with verified digests, making .science exports of long research histories smaller and faster.
☁️ Improved Cloud Compute Authentication
Modal and Hugging Face Jobs now prepare a compatible Python Runtime before browser authorization.
Account verification follows the system proxy, improving connectivity and sign-in reliability.
📚 Literature CSV Export
Literature metadata can now be exported as CSV, alongside BibTeX and RIS.
Researchers can export selected references across pages or complete Projects and Collections using 16 standardized English columns.
【🔧 Improvements】
- Built-in Agent now respects provider-supported output limits and native protocols.
- Improved context usage measurement and compaction in long Sessions.
- Answered Agent questions continue without unnecessary persistence delays.
- Research Film generation continues when switching Sessions.
- Idle Python, R, REPL, and Shell Runtimes are automatically reclaimed after 15 minutes of inactivity.
- Improved Resource Center presentation and context-reset notifications.
【🐛 Bug Fixes】
- Windows & Updates — Fixed signed update packaging, installation blockers, process detection, and recovery diagnostics.
- Agents & Providers — Improved provider error reporting, OpenCode request handling, and Conversation completion.
- Notebook & Compute — Restored Notebook attachment previews, local file links, and Linux Runtime recovery.
- Literature & PDF — Fixed DOI capitalization, reference import routing, damaged PDF extraction, and translation isolation.
- Sessions & Packages — Fixed local CLI package imports, Conversation persistence, and first-message navigation.
- Web & Interface — Improved mobile pairing, remote-control error reporting, and responsive source previews.
1
4
290
Can we reproduce Chinchilla scaling?
AIPOCH Open-Science closely matched Epoch AI’s fit using reconstructed data.
Optimizer settings affected uncertainty; ~20 tokens per parameter remained compatible with the main fit.
Can We Reproduce Chinchilla Scaling?
youtube.com/watch?v=U78VT1-n…
Chinchilla’s “20 tokens per parameter” rule deserves a closer look. With AIPOCH Open-Science, we revisited the public evidence behind compute-optimal AI training and made the analysis available to inspect and replay.
1️⃣ Start with the evidence: 245 points reconstructed by Epoch AI from Chinchilla’s Figure 4. Use 240 for the main fit and retain five unusually undertrained runs for a sensitivity check.
2️⃣ Refit the scaling law: Model loss as a function of parameter count and training tokens, using 4,500 initializations and a robust log-space objective. The result closely matches Epoch AI’s fitted parameters.
3️⃣ Compare on equal terms: On the same 240 observations, the original paper’s rounded Approach 3 parameters produce about 4× the refit’s log-space Huber error.
4️⃣ Investigate uncertainty: Run 2,000 bootstrap resamples and a separate 100-repeat, 80%-subsample analysis. The latter’s 10th–90th percentile interval for the model-size allocation exponent is about 26× wider than originally reported. Optimizer diagnostics show how early stopping can artificially narrow intervals; they do not establish what happened in the original study.
5️⃣ Test the “20” rule: At 10²⁶ FLOP, the main fit projects 15.5 tokens per parameter, with a 95% bootstrap interval of 4.5–42.4. Including all five extreme runs shifts the estimate to 3.5. These are extrapolations far beyond the observed compute range.
6️⃣ Keep the work inspectable: AIPOCH Open-Science preserves the report, six figures, code, data and comparison runs together, including the earlier implementation revised during the investigation.
This reproduces a fit to reconstructed data; it does not retrain Chinchilla.
2
75
🚀 AIPOCH Open-Science v1.2.0 is out! 🎉
This release brings cloud computing, a built-in AI Agent, and more flexible research workflows.
✨ Run approved jobs on Kubernetes, Hugging Face Jobs, Modal & Ray
✨ Built-in Open-Science Agent becomes the default for new setups
✨ Claude Code, OpenCode, Codex, CodeBuddy & Pi remain supported
✨ Faster macOS updates with differential downloads
✨ Step-linked discussions for imported .science research
✨ Cleaner Workspace navigation & Session management
✨ Source Link Previews with metadata
⚡ Improved Notebook, PDF translation & cross-platform stability
AIPOCH Open-Science v1.2.0 — more compute options, smarter Agents, and more connected research workflows.
🔗 github.com/aipoch/open-scien…
⭐ Star us: github.com/aipoch/open-scien…
AIPOCH Open-Science v1.2.0 is out!
github.com/aipoch/open-scien…
This release focuses on cloud computing, AI Agent capabilities, and research workflow continuity. It introduces four new remote compute backends, a built-in Open-Science Agent, faster macOS updates, step-linked discussions for imported research, and improvements to Workspace navigation, Notebook performance, and PDF translation.
【🚀 New Features】
☁️ Four New Remote Compute Backends
Open-Science now supports running approved research jobs on four additional compute platforms:
- Kubernetes
- Hugging Face Jobs
- Modal
- Ray
Alongside existing SSH and Slurm support, researchers can submit jobs with explicit approval of the target, command, resources, and outputs.
Job recovery records also allow result collection to resume after a restart, with published outputs accessible through Compute Preview.
🤖 Built-in Open-Science Agent
Open-Science now includes its own built-in Agent Runtime, which becomes the default for new installations.
- Provider-neutral Agent execution.
- Streaming support across major API protocols.
- Validated Tool handling and persistent execution records.
- Existing Claude Code, OpenCode, Codex, CodeBuddy, and Pi frameworks remain available.
⚡ Faster macOS Updates
macOS now supports differential application updates.
Instead of downloading the entire application, updates retrieve only changed blocks, with automatic fallback to a full download when necessary.
💬 Step-Linked Research Discussions
Imported .science packages and Replay steps now support step-linked child Conversations.
Researchers can ask questions about specific research steps or Notebook Runs while preserving the original Session identity and history.
Discussions remain available even when the parent Session is archived.
🔍 Source Link Preview
Source links now display a preview card with the destination domain and, for approved HTTPS sources, available title and description metadata.
【🔧 Improvements】
- Simplified Workspace Sidebar with Pinned, Imported Research, Active, and Recent Sessions.
- Imported research now highlights Fork to Continue as the primary next action.
- Sending a Prompt from an earlier Conversation turn preserves the reading position.
- Model Settings reorganized into Local Capabilities for PDF extraction, OCR, and research narration.
- Clearer Tool error messages with actionable recovery guidance.
- Improved Notebook responsiveness and large-source processing.
- More accurate scientific PDF text and structure extraction.
- Improved Replay navigation and research video export controls.
【🐛 Bug Fixes】
- Remote Compute & Notebook — Improved Notebook recovery, Windows R execution, Node Runtime selection, and background task handling.
- Agents & Providers — Fixed DeepSeek model routing and Codex Runtime cleanup on Windows.
- PDF & Literature — Fixed PDF translation failures so successfully translated pages remain available when individual pages encounter unsupported layouts.
- Application Updates — Improved macOS and Windows update checks, process detection, and installation recovery.
- Workspace & Interface — Fixed Windows display-scaling issues that could place input controls behind the taskbar.
- Runtime & Recovery — Improved startup diagnostics, backend restart recovery, and Windows Notebook process cleanup.
3
526
Can NVIDIA’s Lyra 2.0 generate better-looking frames while keeping a 3D scene consistent? An AIPOCH Open-Science agent audited all 35 values in the paper’s ablation table across five variants and seven metrics.
youtube.com/shorts/2NiBL2dus…
Explore the Lyra 2.0 ablation in AIPOCH Open-Science. Trace seven metric changes and the reprojection error that changes a comparison.
2
138
AIPOCH Open-Science just scored 97.30% (36/37) on CORE-Bench-Hard⁻ in a local eval with DeepSeek Flash at Max reasoning effort — beating the highest official reference scores reported so far.
GitHub: github.com/aipoch/open-scien…
AIPOCH Open-Science passed 36 of 37 tasks (97.30%) in a local evaluation on AstaBench’s CORE-Bench-Hard⁻ test split, using DeepSeek Flash (deepseek-flash) at Max reasoning effort.
Can an agent reproduce a result from a published study?
On CORE-Bench-Hard⁻, it starts with the study’s code, data, and README. It installs dependencies, runs the analysis in a container, and records the requested results in report.json.
The benchmark’s programmatic scorer checks the reported numbers, text, and lists against reference results. A task counts as correct only when every subquestion passes. The final tally includes retests and records one failure.
The highest scores in our report’s comparison:
• AIPOCH Open-Science / DeepSeek Flash (Max) — 97.30% (local)
• Ai2 ReAct / gpt-5.4-2026-03-05 — 94.59%
• Ai2 ReAct / gpt-5.5-2026-04-23 — 91.89%
• Ai2 ReAct / gemini-3.1-pro-preview and claude-opus-4-6 — 89.19% each
The local score is 2.70 percentage points above the highest official reference in the report’s snapshot, using unrounded proportions. The full chart shows all 25 official references and preserves their tied ranks. Our local result has yet to be submitted to the official leaderboard.
1
132
🚀 AIPOCH Open-Science v1.2.0 is out! 🎉
This release brings cloud computing, a built-in AI Agent, and more flexible research workflows.
AIPOCH Open-Science v1.2.0 — more compute options, smarter Agents, and more connected research workflows.
🔗 github.com/aipoch/open-scien…
AIPOCH Open-Science v1.2.0 is out!
github.com/aipoch/open-scien…
This release focuses on cloud computing, AI Agent capabilities, and research workflow continuity. It introduces four new remote compute backends, a built-in Open-Science Agent, faster macOS updates, step-linked discussions for imported research, and improvements to Workspace navigation, Notebook performance, and PDF translation.
【🚀 New Features】
☁️ Four New Remote Compute Backends
Open-Science now supports running approved research jobs on four additional compute platforms:
- Kubernetes
- Hugging Face Jobs
- Modal
- Ray
Alongside existing SSH and Slurm support, researchers can submit jobs with explicit approval of the target, command, resources, and outputs.
Job recovery records also allow result collection to resume after a restart, with published outputs accessible through Compute Preview.
🤖 Built-in Open-Science Agent
Open-Science now includes its own built-in Agent Runtime, which becomes the default for new installations.
- Provider-neutral Agent execution.
- Streaming support across major API protocols.
- Validated Tool handling and persistent execution records.
- Existing Claude Code, OpenCode, Codex, CodeBuddy, and Pi frameworks remain available.
⚡ Faster macOS Updates
macOS now supports differential application updates.
Instead of downloading the entire application, updates retrieve only changed blocks, with automatic fallback to a full download when necessary.
💬 Step-Linked Research Discussions
Imported .science packages and Replay steps now support step-linked child Conversations.
Researchers can ask questions about specific research steps or Notebook Runs while preserving the original Session identity and history.
Discussions remain available even when the parent Session is archived.
🔍 Source Link Preview
Source links now display a preview card with the destination domain and, for approved HTTPS sources, available title and description metadata.
【🔧 Improvements】
- Simplified Workspace Sidebar with Pinned, Imported Research, Active, and Recent Sessions.
- Imported research now highlights Fork to Continue as the primary next action.
- Sending a Prompt from an earlier Conversation turn preserves the reading position.
- Model Settings reorganized into Local Capabilities for PDF extraction, OCR, and research narration.
- Clearer Tool error messages with actionable recovery guidance.
- Improved Notebook responsiveness and large-source processing.
- More accurate scientific PDF text and structure extraction.
- Improved Replay navigation and research video export controls.
【🐛 Bug Fixes】
- Remote Compute & Notebook — Improved Notebook recovery, Windows R execution, Node Runtime selection, and background task handling.
- Agents & Providers — Fixed DeepSeek model routing and Codex Runtime cleanup on Windows.
- PDF & Literature — Fixed PDF translation failures so successfully translated pages remain available when individual pages encounter unsupported layouts.
- Application Updates — Improved macOS and Windows update checks, process detection, and installation recovery.
- Workspace & Interface — Fixed Windows display-scaling issues that could place input controls behind the taskbar.
- Runtime & Recovery — Improved startup diagnostics, backend restart recovery, and Windows Notebook process cleanup.
1
2
95
Gwayne Stark retweeted
AIPOCH Open-Science passed 36 of 37 tasks (97.30%) in a local evaluation on AstaBench’s CORE-Bench-Hard⁻ test split, using DeepSeek Flash (deepseek-flash) at Max reasoning effort.
Can an agent reproduce a result from a published study?
On CORE-Bench-Hard⁻, it starts with the study’s code, data, and README. It installs dependencies, runs the analysis in a container, and records the requested results in report.json.
The benchmark’s programmatic scorer checks the reported numbers, text, and lists against reference results. A task counts as correct only when every subquestion passes. The final tally includes retests and records one failure.
The highest scores in our report’s comparison:
• AIPOCH Open-Science / DeepSeek Flash (Max) — 97.30% (local)
• Ai2 ReAct / gpt-5.4-2026-03-05 — 94.59%
• Ai2 ReAct / gpt-5.5-2026-04-23 — 91.89%
• Ai2 ReAct / gemini-3.1-pro-preview and claude-opus-4-6 — 89.19% each
The local score is 2.70 percentage points above the highest official reference in the report’s snapshot, using unrounded proportions. The full chart shows all 25 official references and preserves their tied ranks. Our local result has yet to be submitted to the official leaderboard.
Kimi K3.1 Pro open weights: plan before you download.
AIPOCH Open-Science checked the manifest: ~1.56 TB of files. With 2 TB free and a 500 GB reserve, one copy exceeds the budget by ~61 GB.
youtube.com/shorts/d5vtzd9Pf…
Kimi K3.1 Pro Open Weights: Can I Run It on My Computer?
- Why does the answer change with the same 2 TB of free space?
- If the files fit, how long would the transfer take?
Read the full article here: aipoch.com/blog/kimi-k3-1-pr…
4
411
See how AIPOCH Open-Science used the Moonshot file list to estimate disk space and download time before attempting to run Kimi K3.1 Pro on a home computer.
Kimi K3.1 Pro Open Weights: Can I Run It on My Computer?
- Why does the answer change with the same 2 TB of free space?
- If the files fit, how long would the transfer take?
Read the full article here: aipoch.com/blog/kimi-k3-1-pr…
2
53
Weight-Space Programming: did every math problem improve?
AIPOCH Open-Science checked published scores for 25 matched problems: 18 improved, 7 stayed at zero. Recorded correctness rose from 5% to 47%.
youtube.com/shorts/Mn4tPAfbP…
Weight-Space Programming: Did Every Math Problem Improve After the Weight Edit?
- A fair comparison starts with the same problems
- Eighteen improvements still leave seven zero-to-zero ties
- AIPOCH Open-Science connects the headline to inspectable evidence
Read the full article here: aipoch.com/blog/weight-space…
3
123
🚀 AIPOCH Open-Science v1.0.8 is out! 🎉
This release brings smarter literature workflows, cleaner Settings, and better long-session performance.
🔗 github.com/aipoch/open-scien…
⭐ Star us: github.com/aipoch/open-scien…
AIPOCH Open-Science v1.0.8 is out!
This release focuses on literature management, streamlined Settings, and long-session reliability. It introduces batch reference imports, AI-powered study snapshots, improved Agent process cleanup, and fixes across PDF processing, application updates, and Runtime stability.
【🚀 New Features】
⚙️ Redesigned Settings
Settings are now organized into more focused tabs for easier navigation.
- Appearance and Notifications move under System Settings.
- General Settings now includes app information, diagnostics, CLI tools, and help resources.
- Direct access to the manual installation and update guide.
📚 Batch Literature Import
The Literature Library now supports importing up to 100 mixed references at once.
- Supports DOI, arXiv ID, PMID, PMCID, and official article links.
- Automatically normalizes and deduplicates identifiers.
- Preserves the existing review and transactional save workflow.
🤖 AI-Powered Study Snapshots
Literature details now include an AI Study Snapshot generated using the configured main model.
Each snapshot summarizes five key aspects:
- Research Question
- Study Population
- Methods
- Main Findings
- Limitations
Snapshots clearly indicate whether they were generated from the full text or abstract.
⚡ Improved Agent Process Management
Completed Sessions now automatically release Agent and Tool processes across all five supported Agent Frameworks.
Shared Reviewer Runtimes, outdated handoff resources, and failed cleanup attempts are handled more reliably, reducing memory accumulation during long research sessions.
🧠 Updated Model Support
Tencent TokenHub now supports:
- Hy4 reasoning model.
- DeepSeek V4.1 Flash with a 1M-token context window.
- Image input for GLM-5.3-Flash, Kimi K3, and MiniMax M3.
【🔧Improvements】
- eQTL Catalogue now uses indexed files following the retirement of its REST API, with GRCh38 coordinate validation.
- Improved PDF reading order, Table extraction, and source-text verification.
- Clearer application update dialogs and recovery diagnostics.
- Stronger R Notebook dependency tracking for more accurate reproducibility evidence.
- Improved Artifact Provenance reporting and recovery.
【🐛 Bug Fixes】
- Application Updates — Fixed macOS and Windows update preparation, process detection, and file-lock handling.
- Notebook & Runtime — Fixed Windows recovery issues, sandboxed Node.js proxy compatibility, and shared gateway stability.
- Agents & Sessions — Fixed Pi installation in packaged Desktop builds and improved Agent process cleanup.
- PDF & Literature — Fixed Windows PDF worker crashes, stale PDF import notices, and extraction reliability.
- Artifacts & Downloads — Fixed download permission errors, cleanup failures, and outdated Markdown Artifact provenance.
- Interface — Removed duplicate unavailable-model notices and improved error feedback.
1
83
From link reconstruction and verification → dual assessment of intent and topic relevance → automatic curation of a 5-video watchlist, with each selection backed by the original supporting comments.
youtube.com/shorts/LrtqRu4Y-…
HackerTube turns Hacker News discussions into a video-discovery site.
youtube.com/watch?v=FmNkyCTR…
Inspired by Andrej Karpathy’s project, we put a question to AIPOCH Open-Science: can an agent tell when a video is being recommended, cited or criticized?
For this independent study, the agent examined a small astronomy search sample: 23 comments, 36 unique videos and 38 comment–video pairs. It reconstructed the links from the original comments, recovered a plain-text URL, and verified that all 38 pairs matched the supplied mapping.
Then it read each comment alongside the available parent context, assessing recommendation intent separately from astronomy relevance. Across the 38 pairs, it labeled 25 as recommendations and 13 as neutral references. None was labeled as criticism of a video.
One revealing case: a reply challenged an earlier comment’s claim while praising the videos it linked. Following the conversation helped the agent identify what the criticism was actually aimed at.
The separate topic check identified 21 astronomy-related pairs. AIPOCH Open-Science then assembled a five-video watchlist covering beginner observing, astronomy history, Starlink, Pluto and astrophotography, with each choice linked to its supporting comments.
The same agent reviewed the evidence again and refined four explanations without changing the labels or selections. The .science session preserves the source comments, judgments and revisions alongside the watchlist, report, six figures, code and data, so readers can inspect how each choice was made.
3
72
AIPOCH Open-Science achieves 93.33% accuracy on LitQA2-FullText (70/75), the highest among 29 entries in the compared snapshot. Strong result for literature-grounded scientific reasoning.
AIPOCH Open-Science recorded 93.33% accuracy on LitQA2-FullText: 70 correct answers across all 75 questions in the official test split.
Using deepseek-flash at Max reasoning effort, the local final result is numerically highest in this report’s 29-entry comparison—alongside 28 official records from the October 28, 2026 snapshot.
What does this benchmark test?
An agent receives a scientific question and answer options, searches and reads the literature, identifies relevant experimental conditions and specific results, then selects an answer. The task tests whether literature research leads to the correct answer.
This is LitQA2-FullText answer accuracy, distinct from the LitQA2-FullText-Search retrieval metric reported previously.
The highest scores in the report comparison:
- AIPOCH Open-Science / deepseek-flash, Max — 93.33% (local)
- ReAct / claude-opus-4-6 — 90.67%
- ReAct / claude-sonnet-4-6 — 89.33%
- ReAct / gpt-5.4-2026-03-05 — 89.33%
- ReAct / gpt-5.5-2026-04-23 — 89.33%
That puts the local result 2.67 percentage points above the highest official reference in the snapshot, calculated from unrounded question proportions.
How the result was scored:
Asta-Bench v0.5.4’s official score_litqa2 program compares the submitted answer option with the reference answer. No LLM judge is used. Each of the 75 questions contributes one final scored result: 70 correct, 5 receiving no credit, 0 awaiting a score.
The accompanying full chart retains all 29 records and their ties.
This is a local evaluation compared with a historical official snapshot—not a claim of official leaderboard placement. Answer accuracy does not separately assess citation quality or evidence completeness.
2
36
Gwayne Stark retweeted
AIPOCH Open-Science recorded 93.33% accuracy on LitQA2-FullText: 70 correct answers across all 75 questions in the official test split.
Using deepseek-flash at Max reasoning effort, the local final result is numerically highest in this report’s 29-entry comparison—alongside 28 official records from the October 28, 2026 snapshot.
What does this benchmark test?
An agent receives a scientific question and answer options, searches and reads the literature, identifies relevant experimental conditions and specific results, then selects an answer. The task tests whether literature research leads to the correct answer.
This is LitQA2-FullText answer accuracy, distinct from the LitQA2-FullText-Search retrieval metric reported previously.
The highest scores in the report comparison:
- AIPOCH Open-Science / deepseek-flash, Max — 93.33% (local)
- ReAct / claude-opus-4-6 — 90.67%
- ReAct / claude-sonnet-4-6 — 89.33%
- ReAct / gpt-5.4-2026-03-05 — 89.33%
- ReAct / gpt-5.5-2026-04-23 — 89.33%
That puts the local result 2.67 percentage points above the highest official reference in the snapshot, calculated from unrounded question proportions.
How the result was scored:
Asta-Bench v0.5.4’s official score_litqa2 program compares the submitted answer option with the reference answer. No LLM judge is used. Each of the 75 questions contributes one final scored result: 70 correct, 5 receiving no credit, 0 awaiting a score.
The accompanying full chart retains all 29 records and their ties.
This is a local evaluation compared with a historical official snapshot—not a claim of official leaderboard placement. Answer accuracy does not separately assess citation quality or evidence completeness.









