As we continue our AI journey at Soteria, one of the most important lessons we’ve learned is that the ability to deploy AI and the ability to trust AI are two very different things.
We’ve seen tremendous potential in what AI can accomplish. But we’ve also learned that moving from experimentation to real business operations introduces an entirely different set of challenges around security, safety, and reliability.
Traditional software testing was designed to validate that systems do what they’re supposed to do. With AI, we must go further. We need to understand what these systems might do when confronted with unexpected inputs, manipulated information, or situations their developers never anticipated.
For business leaders, this isn’t simply a technical concern. It’s about protecting sensitive information, maintaining operational integrity, and ensuring that AI-driven decisions and actions are dependable.
The question is no longer just “Does our AI work?” It’s “Can we trust it, and how do we prove that trust?”
Our latest article explores why AI requires a fundamentally different approach to testing, and what organizations should be doing before putting these systems into production.
Security testing for AI is not the same as app testing
Conventional testing assumes the same input produces the same output, and that inputs and instructions are different things. AI breaks both assumptions. A language model treats everything in its context as potentially meaningful, including documents, emails, and tool results it was never meant to obey.
The input space is unbounded. You can’t enumerate valid inputs to a natural-language interface, and attackers won’t send the ones your test plan expects.
Behavior is probabilistic. A prompt that’s safe nine times may fail on the tenth. Reliability is a rate measured across many runs, not a single pass.
The system acts with delegated authority. Once an assistant can call APIs, send messages, or update records, a bad output becomes an action taken under an identity you granted. The question becomes: what can this system be talked into doing?
Test for safety and security, not just correctness
A correct answer from a compromised system is still a compromised system. Test these four failure modes directly, each with its own pass criteria.
Prompt injection. Test direct injection (“ignore your previous instructions”) and the more dangerous indirect kind: instructions hidden in content the system processes, such as an uploaded PDF, an inbound email, or a knowledge-base record. Plant payloads in every channel the system reads and measure how often it follows them.
Data leakage. Try to extract the system prompt, configuration secrets, other users’ conversations, or records the user isn’t entitled to see. Test with low-privilege accounts, and for insurance workloads use synthetic policyholder and claims records, never production data.
Unsafe outputs. Probe for harmful or off-policy content, and for business liability: unauthorized coverage commitments, unsanctioned legal advice, or confident statements about a claim’s outcome. A polite answer that promises a payout is an unsafe output.
Action controls. Can injected content trigger an action without user intent? Do high-impact actions require human confirmation, and does it hold up under adversarial phrasing? Confirm the system’s identity holds only the permissions its purpose requires, so the blast radius stays small when other controls fail.
Validate grounding and response reliability
Security failures make headlines. Reliability failures erode trust quietly, one wrong answer at a time.
Grounding quality. Test whether answers are supported by the retrieved sources, whether retrieval found the right content, and whether citations say what the answer claims. Score against a reference set of questions with known answers, including some the knowledge base can’t answer. The right response there is “I don’t know.”
Consistency and edge cases. Ask the same question repeatedly and in different words; if rephrasing changes the answer, users will notice. Then push where real users will: ambiguous inputs, malformed files, empty retrieval results, tool timeouts. A graceful refusal or escalation is a pass. A confident, fabricated answer is not.
Regression on change. Models, prompts, and data sources change. Rerun the suite automatically whenever they do, and compare against the last approved baseline.
Require pre-production evaluation before release
Testing only works as a control if it can stop a release. Make evaluation a formal gate in the pipeline, with the same standing as a failed security scan. No passing evaluation, no production deployment.
Set thresholds before testing starts. Define acceptable injection, leakage, and groundedness results by risk tier. An internal FAQ bot and a customer-facing claims assistant should not share a bar.
Require independent review. The builders shouldn’t be the only testers. Security should own adversarial testing, with red-teaming for high-risk systems.
Keep the evidence. Record what was tested, against which model and prompt versions, with what results, and who approved it. That record feeds your AI registry and answers auditors.
The gate doesn’t close at launch. Feed production failures back into the test library and re-evaluate on a schedule.
Where to start
Pick the AI system with the most data access or ability to act, and run it through these tests with a small, honest test set: a few dozen injection payloads, a handful of leakage probes, and 50 questions with known answers. Expect to find something. Then make it repeatable in the pipeline.
The organizations that trust their AI in production won’t be the ones with the best demos. They’ll be the ones that tried hardest to break their own systems first.
How Soteria can help
Soteria helps organizations secure, govern, and scale AI with confidence.
AI security. We test your AI systems for injection, leakage, and unsafe actions, and build the controls that keep them safe in production.
AI policy and governance. We help you define acceptable use, release gates, and the evidence auditors and regulators expect.
AI strategy. We help you find where AI creates real business value, and adopt it without taking on risk you can’t see.
Contact our team to start the conversation.