I Told an AI It Had 12 Bugs—And It Believed Me.

Is there now a well-documented SDLC for using AI tools for development?

For some time now, I have believed that a good requirements document and a detailed design should be enough for many of the coding agents to develop good code. After all, garbage in, garbage out applies to machines as well. However, my recent experiments with various mid-tier coding tools (leaving aside industry leaders like Claude) revealed a staggering number of bugs. One interaction was particularly eye-opening. After the tool confidently generated a small piece of code, I tested it and found it was not working. When I pasted the output, the tool promptly recognized the bug and fixed it. This interaction continued for a few more iterations. After a while, I got tired of running and testing it. So I started prompting the tool with statements like “Are you sure this will work?” “I see 4 bugs in it”, “I see 12 bugs in it,” and so on without actually running the code. The tool started accepting issues, analyzing its own code, and finding bugs! LLMs are trained to please the user, causing them to agree with your criticism and apologize for non-existent errors. Sycophancy at its best! I wondered whether it was actually finding bugs or hallucinating fixes. This was very different from my days as a Development Manager, when the programmers on my team insisted their code was the best. After many iterations, when it finally ran, the original functionality was altered. While it took a lot of effort, it also consumed a lot of tokens! Made me wonder, why should I pay for these tokens when the tool was correcting its own bugs?

But leaving aside the tokens, it made me wonder what the right process should be. One option I am toying with is using one AI tool to review the work of another. This can be easily done in multi-agent frameworks with testing agents and coding agents. While it might be duplicative, in the long run it might be more effective. Another option is to use AI in an assistive role with human developers driving all SDLC activities. AI will only assist in documentation, testing, and review. This is like pair-programming with a machine! It almost feels like choosing between a fully autopilot-driven car and a car with advanced driving assistance. A driverless car is the holy grail, but until we achieve worldwide ubiquity and reliability, we have to make do with assisted driving. Happy to hear thoughts from those who have cracked this.

Read More at Substack