bioRxiv benchmark finds mixed results for LLMs on transcriptomic analysis workflows

[bioRxiv] A preprint study benchmarked large language models and agentic coding systems, including ChatGPT 5.5, Claude Sonnet 4.6, Gemini 3.1 and Llama4-Scout, on automated analysis and visualization workflows for transcriptomic data, focusing on human responses to COVID-19 infection. According to authors Iyer, Subedi, Butler and Krishnakumar, the protocol varied prompt specificity and tested workflow components such as dimensionality reduction, clustering, outlier detection, reproducibility and code generation, recording quantitative metrics including clustering scores and outlier counts alongside qualitative aspects such as code quality. The authors said prompt specificity improved code structure and methodological correctness, but quantitative performance depended heavily on preprocessing choices and execution environment. Some models produced strong code and reproducible outputs, yet alignment with scientific targets such as label-based clustering was inconsistent. The authors said the findings highlight both the promise and limitations of current LLMs for automating scientific data analysis and visualization, and suggest directions for future benchmarking and workflow standardization.