A simulated fruit fly brain tested the code and got 88% correct Nick Nissi, an engineer at WorkOS, took Google's map of a male fruit fly brain, all 176,422 neurons, and ran it on a laptop. A fly can't read a code. So a helper AI answers 30 yes/no questions about each change, and the answers turn into a smell. A cloned bug tastes bitter, a harmless change tastes like sugar. If the smell causes the fly to stick its tongue out, it approves. Of 301 changes he had never seen, the fly was right 88.3% of the time. Coin flip gets 50%. A harder test went wrong. Given 24 tasks that were supposed to change what the code was doing, it didn't commit any of the correct changes. He studied "has the behavior changed", not "has this change been true". "A basic statistical classifier reading the same scent scored 94.8%. You can try the fly as per your pull request.

A simulated fruit fly brain tested the code and got 88% correct Nick Nissi, an engineer at WorkOS, took Google's map of a male fruit fly brain, all 176,422 neurons, and ran it on a laptop. A fly can't read a code. So a helper AI answers 30 yes/no questions about each change, and the answers turn into a smell. A cloned bug tastes bitter, a harmless change tastes like sugar. If the smell causes the fly to stick its tongue out, it approves. Of 301 changes he had never seen, the fly was right 88.3% of the time. Coin flip gets 50%. A harder test went wrong. Given 24 tasks that were supposed to change what the code was doing, it didn't commit any of the correct changes. He studied "has the behavior changed", not "has this change been true". "A basic statistical classifier reading the same scent scored 94.8%. You can try the fly as per your pull request.