Why we are doing this

Most of the AI in your life, you never chose, and you can't turn off.

It is already in use

Epic's sepsis model scores hospital patients continuously from their records. Insurers use automated review to deny payment. Banks, landlords, employers and the government score you with systems you never chose. We counted 56 of these systems. You cannot opt out of 38. Nine let you opt out in name only. Nine actually let you say no.

Nobody tested it

If people haven't seen lots of CEOs sharing this, the problem isn't the testing. The problem is people have skipped the testing. It shouldn't be a clinical company working on maternal mortality being the cutting edge of AI use failure.

Competence doesn't cascade

All of adult learning runs on graduated responsibility. This tool is not that. It can be brilliant at something very complex and completely wrong at the simplest thing underneath it. So you have to check it. Every output. Every "done." They built an entire economic model that requires these systems to run without a human in the loop. And these systems, as currently constituted, should not.

Nobody is responsible when it fails

Trust is track record plus skin in the game. The machine has neither. You cannot deploy a tool like this into medicine before you've figured out who is responsible when it fails. And nobody has.