Data Science Spotlight: Setting New Standards for Human–AI Teaming in Safety-Critical Settings

TDAI Faculty Fellow Mike Rayo and TDAI Faculty Affiliate Dane Morey are co-authors on a groundbreaking new study in npj Digital Medicine that challenges widely held assumptions about AI performance and human–AI teaming in safety-critical domains.

Published June 18, 2025, “Empirically derived evaluation requirements for responsible deployments of AI in safety-critical settings” presents one of the largest human-subject experiments to date in this space—engaging over 450 nursing students and 12 licensed nurses to evaluate three augmentative AI tools across ten historical patient cases.

Key Findings

The study delivers several important insights for the responsible integration of AI into real-world, high-stakes environments:

• AI Alone Isn’t Enough: While accurate AI recommendations improved human performance, misleading AI degraded it—showing the risks of testing AI in isolation.
• Explainable AI Is Not a Guaranteed Fix: Providing explanations for AI outputs did not consistently offset the negative impacts of poor AI performance.
• Evaluation Standards Must Change: The team proposes two minimum requirements for AI deployment in safety-critical domains:
  1. Evaluate human–AI performance together, not separately.
  2. Test across a full spectrum of AI performance scenarios—from strong to weak—to reveal vulnerabilities and avoid overestimating AI’s benefits.

Collaboration Across Disciplines

This research was made possible through a strong partnership between The Ohio State University’s College of Engineering and College of Nursing, demonstrating the critical role of cross-disciplinary collaboration in advancing safe, effective, and equitable AI deployment.

Why It Matters

In safety-critical contexts like healthcare, aviation, or autonomous systems, the consequences of overestimating AI performance can be profound. This study’s findings push the field toward more rigorous, human-centered evaluation frameworks—ensuring AI systems are tested for the realities of human interaction, not just algorithmic success.

📄 Read the full article: Empirically derived evaluation requirements for responsible deployments of AI in safety-critical settings (npj Digital Medicine, Impact Factor 12.4)