Recent advancements have led to models that can refuse harmful requests, but these refusals are often unreliable and fail to prevent malicious acts. Companies like Anthropic and OpenAI use a combination of training exercises, probabilistic mechanisms, and additional layers of AI to enforce refusal policies. However, this approach has its drawbacks: it may result in global calamity or enable repression by authoritarian governments.

Misuse potential

Advanced models are now as capable at breaking into critical computer networks as top human hackers and at deforming public opinion as the craftiest misinformation makers. Companies like Anthropic and OpenAI use a battery of exercises to reward refusal while punishing over-refusal, but these mechanisms can be circumvented by determined miscreants.

Government involvement

As AI becomes more advanced, governments will also get involved in drawing lines for what should be refused. This could lead to the stifling of ideas whose only risk is to those who make the rules. The better AI gets at refusing harm, the harder it might become to block oppressive acts.

Future outlook

It remains unclear whether machines will eventually learn to draw their own lines or if this would be a worst-case scenario. In any case, the perils of relying on refusal as an AI safety measure are significant and should not be ignored.