When an autonomous robot makes a mistake, like misidentifying an object or taking an unsafe path, who’s accountable, and why did it happen? These questions are becoming increasingly harder to answer with the rise of foundation models and generative AI in robotics. Unlike rule-based systems, large-scale models can sometimes behave unpredictably. That’s why transparency and accountability can’t be afterthoughts; they must be built into how these systems are designed, trained, and deployed from the start.

As AI-powered robotic systems become more capable and autonomous, they’re being deployed in increasingly high-stakes environments: factory floors, warehouses, and farms. These machines don’t just process information; they act in the physical world, raising urgent questions about safety, transparency, and accountability.

We spoke with Dr. Ransalu Senanayake, Assistant Professor of Computer Science at Arizona State University, to better understand how we build transparency and accountability into autonomy.

Image source: Ransalu Senanayake, Som Sagar, Jiafei Duan, Sreevishakh Vasudevan, Yifan Zhou, Heni Ben Amor, and Dieter Fox. “From Mystery to Mastery: Failure Diagnosis for Improving Manipulation Policies.” Published February 8, 2025.

Transparency and Accountability Explained

Ashutosh Saxena: How should we think about transparency and traceability in robotics systems powered by large-scale models, and why are these concepts so critical as AI becomes more complex and autonomous?

Ransalu Senanayake: First, let’s define transparency. Transparency refers to the ability to understand, explain, and inspect how a robotic system makes decisions and takes actions. This becomes particularly important as large-scale generative models are integrated into robotics, introducing new layers of abstraction and unpredictability. Transparency is essential in robotics because these systems operate in the physical world, often in unpredictable environments, where their actions can have real-world consequences.

When it comes to traceability, especially in large, complex robotic AI implementations, traceability refers to the ability to identify which component is responsible if something goes wrong. For example, imagine a robot that uses a camera to detect an object and then tries to pick it up with a robotic arm. If the action fails—let’s say it drops the object—how do we know what went wrong? Did the robotic arm’s control policy, the “decision-making brain,” fail? Was it the motor? Or was it the perception system that misidentified the object in the first place?

Traceability helps us break the system down and isolate the source of the failure. In these complex, interconnected systems, it’s essential for debugging, improving reliability, and assigning accountability.

But with AI, especially large models like ChatGPT or those used in robotics, there isn’t always a clear, interpretable path. If you type a question into ChatGPT, no one can point to a specific line of code and say, “This is why it responded that way.” The same applies to many robotic systems using neural networks. So in that sense, traceability in AI and robotics is a much more complex and evolving challenge.

Ashutosh Saxena:  Thanks. Now that we have a better understanding of what transparency means in today’s robotics systems, especially those powered by foundation models and generative AI. In your field of expertise, how would you define accountability?

Ransalu Senanayake: Accountability means that when a robot malfunctions, underperforms, or fails to complete a task successfully, there is a transparent and traceable way to determine who is responsible. This becomes especially challenging with foundation models and generative AI, which often operate as opaque, large-scale systems with complex behaviors that may not be easily explained or anticipated.

Whether it’s the developer, manufacturer, integrator, or operator, accountability implies that:

• The system includes mechanisms for auditing and reviewing actions, particularly critical for generative models with emergent behavior
• Responsible parties are held to ethical, legal, or regulatory standards
• Failures can be diagnosed and corrected, with lessons learned from both the data and the model’s outputs
• There is a framework for legal liability, operational responsibility, and recourse, even when large-scale AI components make decisions

Accountability ensures that autonomy doesn’t mean ungoverned. As generative AI becomes central to robotic decision-making, someone remains answerable when things go wrong. At the same time, it’s important to ensure that accountability measures support innovation and don’t hinder the development of new AI technologies.

Oversight at Scale: The Challenge of Governing Intelligent Machines

Ashutosh Saxena: What are the biggest challenges in applying transparency and accountability frameworks to robotics, compared to more traditional software systems?

Ransalu Senanayake: Today, most robots operate in controlled settings like labs, test sites, or tightly managed pilots. But that’s starting to change as more systems are deployed in real-world industrial and public environments.

This shift makes transparency and accountability far more critical. Unlike a search engine like Google, where a wrong result is inconvenient but rarely harmful, robots operate in physical spaces—often around people—where mistakes can lead to injury, property damage, or worse. These systems must be reliable, and we need precise mechanisms to understand and evaluate their decisions.

Applying transparency and accountability in robotics is much more complex than traditional software. With a language model like ChatGPT, you can test inputs and review outputs in a low-stakes environment. But in robotics, testing requires physical environments interactions. That adds cost, complexity, and risk—so the challenge isn’t just technical; it’s operational and resource-intensive too.

Ashutosh Saxena: As industrial robots move from labs into real-world settings like farms and warehouses, are there emerging frameworks for building trust in AI behavior, and how does traceability factor into that?

Ransalu Senanayake: Traditional ISO standards cover general safety for industrial robots, but don’t address AI-specific behavior or trust. There’s no comprehensive framework yet, though organizations like NIST are starting to explore it. The challenge is that standards must be application-specific—what works in a warehouse might not apply to agriculture. As robotics becomes more present in the real world, we need flexible, adaptable guidelines. A key part of that is traceability: when something goes wrong, we must be able to trace it back, log decisions, validate behaviors, and understand precisely what happened. Without that, it’s hard to build trust or assign responsibility.

Image source: Ransalu Senanayake, Som Sagar, and Aditya Taparia. “Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models.” Published June 13, 2024.

Why Validating Large AI Models in Robotics Is So Hard

Ashutosh Saxena: When using large models for robotics, what are the nuances of validating them in real-world settings?

Ransalu Senanayake: In traditional engineering, a well-defined mathematical model allows us to predict how the system will behave. Engineers understand the formulas and parameters, so we can say, “This will work under these conditions” with a high degree of confidence.

But neural networks are very different. These models are essentially massive black boxes; no one fully understands how every part works, similar to the human brain. Just like the brain has billions of neurons, neural networks have millions or even billions of parameters. So, inspecting them line by line or individually is impractical to understand what’s happening.

That’s why we need alternative ways to validate them. These include techniques like probing internal behavior, stress-testing the models, and—critically—evaluating how they perform under different conditions in the real world. But again, doing that in robotics is costly and complicated because it requires physical testing, not just digital simulation.

Image source: Ransalu Senanayake, Aditya Taparia, and Som Sagar. “Explainable Concept Generation through Vision-Language Preference Learning.” Published June 5, 2025.

Auditing and Oversight Mechanisms for Large-Scale Models

Ashutosh Saxena: As robotics systems become more complex, what approaches prove most effective for auditing and verifying their behavior? And what new tools or techniques are most promising for improving transparency, safety, and control, especially in systems built on large neural networks?

Ransalu Senanayake: So this kind of work often goes by different names—we sometimes call it auditing, and in other contexts, it’s called red teaming. The idea is to evaluate the system from an external perspective, to stress-test it under different scenarios, and see how it behaves.

Traditionally, in well-defined software systems, we could apply formal methods—mathematical techniques that allow us to prove system properties with a high degree of confidence. But that approach, despite various efforts, doesn’t translate well to modern AI systems, especially those based on large neural networks.

These systems aren’t purely predictive but non-deterministic in ways that make formal guarantees difficult. Their outputs can be unpredictable, and the architecture is often a monolithic block rather than a modular, inspectable pipeline.

So instead of validating them through rigid rules, we need to rely on empirical testing that can scale to real-world environments. Since we can’t break them down into neat, testable modules, we have to probe them from the outside—stress them, observe how they respond, and develop new ways to verify their behavior through experimentation rather than strict specification. One promising direction my lab has worked on is using deep reinforcement learning as a scalable way to identify failure modes in robotic systems. This kind of testing helps uncover how systems might behave under less-than-ideal conditions.

Regarding transparency and interpretability, there’s a growing interest in post hoc explanation methods. Rather than making the entire system interpretable from the start—which is very difficult with deep learning models—we focus on analyzing and understanding behavior after a system takes action. In other words, when something goes wrong, can we look back and explain why it happened? That kind of reactive interpretability is becoming more practical and useful in real deployments.

‍

Why Responsible AI in Robotics Can’t Wait

As robotic systems become more intelligent and autonomous, the stakes are no longer just theoretical; they’re tangible, immediate, and deeply human. Transparency and accountability must evolve in step with capability, not lag behind it. From model interpretability and real-time traceability to cross-industry collaboration and regulatory foresight, the path forward requires more than technical innovation—it demands a cultural shift toward responsibility.

Dr. Ransalu Senanayake notes, “There’s a lot of work on developing new models for how robots should behave and improve over time—but far less on what it takes to bring those models into the real world. For robots to be deployed safely and ethically, they must meet real-world requirements: safety, fairness, bias mitigation, accuracy, and traceability. That’s the gap we’re working to bridge. But it has to be a much broader, collective effort.”

Ultimately, the future of robotics isn’t just about what machines can do but also about how responsibly we allow them to do it.

Ashutosh Saxena: Ransalu, thank you so much for taking the time to share your insights today. Your perspective on traceability, interpretability, and real-world validation in robotics brings essential clarity to a rapidly evolving space.

Whether you’re dealing with dynamic environments, moving objects, or difficult weather conditions, TorqueAGI is ready to add even more transparency and accountability to your robotic stack. Contact us for a demo to see how we can help!

Image source: Ransalu Senanayake, Som Sagar, Aditya Taparia, Harsh Mankodiya, Pranav Bidare, and Yifan Zhou.. “Trustworthy Conceptual Explanations for Neural Networks in Robot Decision-Making.” Published September 16, 2024.

‍