Self-replicating prompt injections exist · OpenAI Alignment

Research on aligning AI with human values and intent, and reports documenting model failures. We show the existence of a new variety of prompt injection, which can self-propagate akin to a computer worm. No impact was observed outside of the simulated tool calls in training and evaluation; we are sharing this due to the novel nature of the prompt injection, not because of any incident.

Source: alignment.openai.com