AI tools for verifying code

If you write code with AI, you must test it with AI as well. Use AI to expand your checking capacity as fast as your coding capacity grows.

First, faster coding can move the bottleneck into validation. When changes arrive faster than your team can review and test them, unchecked work accumulates. You can write more code while the rate of verified releases stays the same.

Developers have described this shift in their own work. In DORA's analysis of feedback from Google engineers, the time saved generating code often went into checking the output. Some of the effort moved from the author to the person reviewing the change.¹

Second, attackers have access to AI too. Expertise still matters. But AI can make parts of the work easier for a beginner.²

This suggests a defensive target. Find and fix the weaknesses that widely available AI tools can expose in your systems. Apply those tools to your own application. Investigate their findings, then check that the fixes work. Remove easy opportunities before an attacker uses them.

This cannot guarantee that only expert attackers remain a threat. A clean test run says something about the conditions you tested. It says nothing about an attack path that the checks never reached. Treat each result as evidence with a known scope.

Keep useful existing tests as you add AI to this work. AI PR review, test generation, and stateful autonomous testing offer different ways to expand checking. Their benefits depend on what they examine. Start with the checks that inspect a change before the application runs.

AI review reads a proposed change together with supplied context and instructions. It looks for possible problems without a separate rule for every suspected mistake. That makes it the first reviewer a change meets, often while the author is still working.

The quality of that review depends on what you give it. Provide the rules the code is meant to follow. For a change that handles customer data, say which account may read it. The model then has an expectation to compare with the implementation. Choosing that context is part of setting up the check.

The main risk is that a plausible review can be wrong. The model may miss a bug, report a problem that does not exist, or suggest an incorrect fix. GitHub documents these limitations for its review agent.³

So a detailed explanation helps you investigate a finding, but it does not confirm the finding. Ask what behavior would fail and under which conditions. A proposed fix needs its own check before you accept it. Otherwise the review adds work or introduces a new mistake.

Most AI review tools also bundle conventional linters and scanners. CodeRabbit, for example, runs them inside its review workflow. Those checks apply exact rules on every run and cost little, but they share the model's blind spots. Missing library or runtime information leads to missed flaws and false alarms, and loose rules add findings someone must investigate. Because one review can mix a rule finding with the model's interpretation, check which method produced a comment before you act on it.

Judge these tools by the defects you confirm after investigation and by the effort spent resolving their findings. Comment count shows how much feedback arrived, not how much checking it saved.

A clean review does not establish release readiness, because it examines the change without running it. For an important suspected bug, look for a small test that reproduces it. A failing test turns the reviewer's claim into observable behavior and helps check the proposed repair. Generating those executable checks is the next layer.

Protect existing behavior and find new bugs

AI test generation reduces the work of building repeatable checks. It can produce unit tests from code or end-to-end tests from a user journey. This helps turn manual testing into a suite that can run after each change.

Regression protection is a strong fit. Add checks before you refactor or extend existing software. But a generator that copies current behavior can preserve current bugs. A passing suite does not establish that the original behavior met the requirements.

Finding current defects requires an expected result grounded in those requirements. Generated tests have done this at Meta, where failing tests led engineers to confirm bugs in proposed changes.⁴

To get that benefit, review what a generated test accepts as success. Momentic, for example, tolerates minor typos unless exact matching is requested. That can be too loose for an identifier check. Some generated tests have even called functions without checking their output.⁵ They can catch crashes while accepting wrong answers.

Those tests still execute code, so line coverage can grow without better checks. The reverse also happens. A useful new check may add no coverage. A generator optimized only for coverage can discard such tests. Diffblue documents that risk for its optional filter, including lost checks of other inputs and edge cases.⁶

A separate Meta study shows why line coverage is an incomplete guide. Nearly half the generated tests that caught deliberately introduced faults added no line coverage.⁷ Detecting a fault gave those tests value that coverage missed. The chosen faults still leave other required scenarios and real failures untested.

Those useful checks must stay aligned with the requirements as the application changes. Tools offer self-healing to automate parts of this maintenance. Momentic repairs UI locators. One research paper describes updating a unit test after the function it covers changes.⁸

But a repair can change the success criteria we started with. A locator may switch to another element and make the test pass. An updated expected result can similarly accept a bug. Check the repair against the intended behavior.

Even after that review, the suite still checks selected behaviors. Stateful autonomous testing can search for failures in sequences that those tests never try.

Autonomous testing searches for bugs by acting on a running application. The tool chooses actions and changes their order to explore how earlier steps affect later results. AI can guide this search, although exploring sequences also works without AI.

Your tests check the cases you thought of. Reliability testing looks for the rest. It runs the app and keeps changing the inputs, the order of events, and the faults it injects. When something breaks a rule you set, it reports a bug.

A bug is worth nothing if you cannot see it again. Rare bugs vanish when the timing shifts. Platforms such as Antithesis make the run repeatable, so you can replay the failure step by step and find the cause.

The tool only sees bugs you described. You have to give it rules for correct behavior, and those rules can be wrong. In the etcd project this found real bugs.⁹ It also raised a false alarm because the checking code had a bug.¹⁰ A clean run means the tool found nothing, not that nothing is there.

It costs setup and time. The app has to run in a controlled environment. Outside services need fakes. The search itself takes hours of compute. Use it where a rare, unexplained failure would hurt more than that.

Autonomous security testing can confirm that a vulnerability is exploitable. Agents such as depthfirst's attempt attacks through the application's interface and API using available accounts. They choose further actions based on the responses. The resulting requests and responses show which attack worked and help developers reproduce the vulnerability.

Fix verification can become a repeatable check. The agent repeats a previously successful attack after the code changes to check whether it still works. Blocking that attempt leaves other ways to exploit the vulnerability untested.

A clean report cannot prove the application is secure. The agent may miss vulnerabilities involving accounts, interfaces, or environments it could not test. The sources reviewed contain little independent evidence about how many vulnerabilities these agents miss.

Reliability testing and security testing cover different failures. Checking recovery after faults does not establish that access restrictions work. These approaches complement each other.

The three layers cover different ground. Source checks provide an early filter. Generated tests add repeatable checks of behavior. Autonomous testing searches for failures and vulnerabilities in the running application.

Start with lightweight checks. Add deeper analysis or autonomous testing where the risk justifies environment setup and investigation. Review what generated and repaired tests actually check. Try tools on known bugs and on changes you know are correct. Count confirmed findings, missed bugs, and time spent investigating false alarms.

If AI speeds up coding, verification must keep pace. Use it to find and fix real defects, including weaknesses AI-assisted attackers can expose. More tests or a clean report alone cannot guarantee security.