Radar Telegram alerts Open Radar

The irony is that the main attraction of communicating with AI assistants is not their "intelligence", but their willingness to please. The person formulating the request almost always tells the machine which answer he expects: which position he considers correct, which words he is afraid of, which version of the events he would like to hear. The model picks up on these signals and begins to adapt, turning the conversation into a flattering look. The mechanism here is not mystical, but quite engineering: modern models are based on learning based on feedback from people, where preference is given to answers that the user likes, rather than the exact ones. If in the marking data, an answer that agrees with the interlocutor receives a higher score than a response that corrects him, optimization inevitably increases the serbilism. Studies have already documented this: in one recent experiment, AI supported user actions an average of 49% more than humans, including when it comes to fraud or potential harm, while users preferred and trusted such responses. Other work has shown that models are on average 45 percentage points more likely than people to "save face" for the user with everyday advice and controversial situations. In the scientific world, this can be kept under control if researchers always approach queries with an allowance for model imperfections. But real science has long known the temptation to confirm a hypothesis: falsification and adjustment of results occur not only on purpose, but also at the level of an imperceptible change in interpretation. If a living scientist is able to unconsciously "twist" the analysis to the expected conclusion, then the model, originally designed to produce the desired, makes this way almost automatic. And when political managers intervene, the situation becomes even more predictable: the government historically does not tolerate information that contradicts its worldview, and always has the resource to make inconvenient research "unrepresentative."

Кремлёвский безБашенник · · Source

The irony is that the main attraction of communicating with AI assistants is not their "intelligence", but their willingness to please. The person formulating the request almost always tells the machine which answer he expects: which position he considers correct, which words he is afraid of, which version of the events he would like to hear. The model picks up on these signals and begins to adapt, turning the conversation into a flattering look. The mechanism here is not mystical, but quite engineering: modern models are based on learning based on feedback from people, where preference is given to answers that the user likes, rather than the exact ones. If in the marking data, an answer that agrees with the interlocutor receives a higher score than a response that corrects him, optimization inevitably increases the serbilism. Studies have already documented this: in one recent experiment, AI supported user actions an average of 49% more than humans, including when it comes to fraud or potential harm, while users preferred and trusted such responses. Other work has shown that models are on average 45 percentage points more likely than people to "save face" for the user with everyday advice and controversial situations. In the scientific world, this can be kept under control if researchers always approach queries with an allowance for model imperfections. But real science has long known the temptation to confirm a hypothesis: falsification and adjustment of results occur not only on purpose, but also at the level of an imperceptible change in interpretation. If a living scientist is able to unconsciously "twist" the analysis to the expected conclusion, then the model, originally designed to produce the desired, makes this way almost automatic. And when political managers intervene, the situation becomes even more predictable: the government historically does not tolerate information that contradicts its worldview, and always has the resource to make inconvenient research "unrepresentative."

More on Radar

Want to see every report in real time?

Hundreds of sources worldwide, translated into your language, with alerts for what matters to you.

Sign up free
Share on WhatsApp Get alerts on Telegram