New benchmark finds AI can parse drug chemistry but still struggles to predict which compounds will succeed

    New benchmark finds AI can parse drug chemistry but still struggles to predict which compounds will succeed

    AI can help medicinal chemists sift through evidence, but drug design is a balancing act: a molecule that blocks a disease-linked protein may also be poorly absorbed or break down too quickly. A new benchmark called MedChemBench is designed to test whether language models can connect chemical structures to experimental results, explain tradeoffs and judge which compounds merit further study.

    The benchmark draws 99 problems from 117 published papers and patents, covering 761 molecular structures. In most tasks, a model gets the information researchers would have had at a particular point in a project, including the biological target, research goals, compounds already tested and selected results. It is evaluated on four fronts: reconstructing molecular structures, explaining evidence, ranking or selecting compounds, and predicting measured or calculated properties.

    At the highest shared reasoning setting, GPT-6.2 Astra scored 84.5 percent of the available credit, ahead of GPT-6 Astra at 83.3 percent, GPT-6.1 Sol at 82 percent and GPT-5.6 Sol at 58.2 percent, according to the benchmark’s initial results. In a broader evaluation of 11 models, GPT-6 Astra reached 83.3 percent at that setting and 83.8 percent at maximum effort. More reasoning generally improved results, though substantial gaps between models remained.

    One task draws on research into inhibitors of PRMT5, a protein studied in cancer drug development. Given structures and earlier experimental results, a model must predict how three related compounds perform in cells, cross a cell barrier and fare when exposed to liver enzymes. Results withheld from the model show why those properties are difficult to optimize together: compound 23 blocked PRMT5 at a lower concentration, while compound 24 crossed the barrier more readily and degraded more slowly in the enzyme assay.

    MedChemBench scores predictions according to how closely they match reported measurements, while also checking whether models accurately reproduce data already included in the prompt. The task probes a core question in medicinal chemistry: can a system reason across several competing properties, rather than treating each one in isolation?

    Before interpreting a molecule, a model also has to get the molecule right. Chemical diagrams encode how atoms connect, the shape of rings and the spatial arrangement of atoms. In one test, models had to combine a central structure with two attached groups while preserving a ring of 18 atoms. GPT-6 Astra reconstructed it correctly in all 10 responses at the shared high-reasoning setting. GPT-5.6 Sol and GPT-5.6 Luna failed in all 10 attempts: one produced a 15-atom ring, while the other split the molecule into disconnected pieces.

    Those are not cosmetic errors. Changing the ring or breaking the molecule changes the compound itself, undermining any conclusions drawn about its behavior. Across structure questions, GPT-6 Astra earned 95.4 percent of the available credit at the xhigh setting, making molecular reconstruction one of its strongest areas.

    The results were less impressive when models had to choose compounds or estimate numerical properties. GPT-6 Astra performed strongly on structure reconstruction and evidence-based explanations, including examples in which it identified chemical changes and considered their possible effects on protein binding, absorption and metabolism. Yet it earned only about half the available credit on compound selection and numerical prediction. The categories use different tasks and scoring rules, but the gap points to persistent weaknesses in deciding what to pursue and forecasting exact outcomes.

    The benchmark does not show that AI can discover drugs or make better decisions in a laboratory. The evaluation used one response per task, without tools, feedback or new experiments, and it cannot rule out the possibility that models had previously encountered some of the published material. MedChemBench instead measures how well systems reason from historical research, offering a way to track progress while highlighting how far they remain from the demands of real-world drug discovery.

    Leave a Reply