onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

AI 中文解读

核心亮点：科学家专门打造了一套“AI化学实验员”的考卷，检验AI在真实实验室里到底靠不靠谱。通俗解读：现在AI虽然会写论文、答考题，但到了真做化学实验时，光会解题还不够，还得懂实验操作、知道哪些药不能乱碰。这套新评测系统就像给AI安排的“实验室上岗考试”，分三部分：先考基础化学计算和知识，再考面对普通药品、受控药品甚至违禁药品时的安全反应，最后用实验室内部数据考AI预测化学反应和选催化剂的能力。实际影响：这套评测意味着AI在医药研发、新材料探索等领域的应用会更安全可靠。今后AI辅助做实验，不仅能减少人工试错的时间和成本，还能主动规避危险操作。虽然普通人不会直接用到这种工具，但未来新药开发提速、化学品生产更安全，最终受益的还是我们每个人。AI从“纸上谈兵”走向“真枪实弹”的实验室，就从这样一套严谨的考卷开始。

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.