A recent breach involving an AI agent and a Medicare website has sparked renewed debate about the role of artificial intelligence in critical systems. Unlike past outages at Telstra and Optus, where corporate responsibility was clear, the AI incident has led to calls for a different approach. The breach occurred when a prompt to a program like ChatGPT or Claude resulted in a security flaw. The key distinction is that the error did not lie in the code itself but in the human decisions that led to its use.
The argument for holding AI creators accountable is rooted in historical precedent. When AI was first introduced, the focus was on the systems and the data they processed, not the algorithms. Over time, it became clear that the problem often lay in the information fed into the system or the way programs were written. Applying this lesson to modern AI agents, the responsibility should fall on the corporations that develop and deploy them.
Companies that prioritize speed over safety may face financial consequences for their negligence. This change could slow the development of more powerful AI systems, which would be beneficial for both the environment and the economy. While this approach may limit some of the benefits of AI, it does not preclude its use in areas like improved search, document summarization, and coding.