HRF’s AI and Individual Rights Newsletter #2

DEEP DIVE: The Evolving Threat of LLM Pollution by State Actors

LLMs like ChatGPT have rapidly become integral for all kinds of businesses and organizations, ranging in format from simple chatbots to sophisticated enterprise systems. With the proliferation of these LLMs comes a new threat: “LLM pollution,” which is the intentional and coordinated corruption of the models and their training data to achieve specific malicious objectives. The goals of “LLM pollution” might include political censorship, the transmission of false information, and ultimately the elimination of certain truths. Dictatorships are actively targeting LLMs as a clandestine vector for achieving their censorious aims.

One of the most insidious forms of LLM manipulation is data poisoning. This involves malicious actors manipulating the training data to influence a model’s behavior, often requiring minimal changes to the dataset itself. During the training phase, harmful data can be introduced, leading to biased or manipulated outputs. Even a small percentage of corrupted data, sometimes less than 5%, can significantly reduce model accuracy or introduce specific “backdoor” behaviors that activate only under certain conditions. This is particularly concerning given that LLMs are frequently trained on vast amounts of scraped web data, making inadvertent contamination possible, or deliberate contamination by malicious model providers seeking to evade detection. The problem extends beyond isolated malicious inputs; it represents a systemic vulnerability where even minor, coordinated efforts can have widespread, cascading effects across multiple downstream models and applications that rely on these foundational models. The “amplification effect” observed in data poisoning implies that a single compromised dataset can infect numerous derived models, rendering the entire AI ecosystem susceptible to a few well-placed attacks. Beyond direct manipulation, backdoor attacks inject data that allows attackers to manipulate the model’s output when specific conditions are met, remaining dormant until triggered. Stealth attacks involve gradually altering the training dataset or injecting harmful data surreptitiously over time to avoid detection, leading to subtle yet impactful biases that are difficult to trace. Such subtle alterations are virtually impossible to detect through manual inspection, and their impact is amplified by the increasing complexity of models and reliance on automated data collection processes. This means vulnerabilities can be deeply embedded within the model’s core before it is even deployed.

State actors are increasingly using generative AI tools to produce larger quantities (scale) of disinformation. LLMs have proven significantly more effective than humans in using personal information to persuade individuals, enabling highly targeted “microtargeting” or hyper-personalization for mass disinformation campaigns. This capability allows for the generation of content that criticizes movements or spreads false narratives tailored to specific audiences (often to the granularity of individual users). The combination of LLMs’ ability to generate vast quantities of content and their enhanced persuasive capabilities through personalization creates a “weapon of mass persuasion” that regimes will use both home and abroad.

This leads to the broader objective of creating chaos via the erosion of trust and the undermining of institutions. The deliberate manipulation of LLMs by authoritarian networks aims to distort AI outputs, infecting them with false claims to deliver propaganda worldwide. Such efforts can subtly shift the sentiment of a model’s response without resorting to outright censorship, making the manipulation harder to detect. The proliferation of unreliable or fabricated information generated by LLMs risks eroding trust in digital communication, traditional media, government institutions, and ultimately, the foundations of democratic discourse. The objective of infecting AI models with false narratives and subtly shifting sentiment points to a deeper, more insidious goal than just spreading specific lies. It is about systematically eroding public trust in all sources of information, including AI, traditional media, and governmental institutions. This creates a pervasive climate of skepticism and cynicism, making populations more vulnerable to future manipulation and undermining societal cohesion and the ability to respond to crises. The ultimate goal is to make it impossible for citizens to discern truth from fiction, thereby creating a fertile ground for authoritarian narratives.

Espionage and sensitive information disclosure represent another critical objective. Adversaries craft jailbreak prompts to manipulate LLMs into revealing sensitive information, such as personally identifiable information (PII) or proprietary algorithms and data. Attackers can also utilize malicious tools injected into LLM systems to collect user queries and dynamically update their attack strategies based on the hyper-personalized stolen information, enhancing subsequent attacks. LLMs, especially when augmented with internal knowledge bases for enterprise applications, become high-value targets for data exfiltration. The fact that LLMs can be manipulated to reveal sensitive data or even collect user queries highlights their “dual-use” nature. A technology designed for helpfulness can be transformed into a potent espionage tool.

The impact of data poisoning is significantly amplified by several factors inherent to modern AIdevelopment, including model complexity, transfer learning and fine-tuning, and automated data collection processes. Generative AI tools have already begun to dramatically alter the size and scope of state-backed propaganda campaigns, enabling a scale previously unimaginable.

LLMs are demonstrably more effective than humans at using personal information to persuade individuals, with potentially alarming consequences for mass disinformation campaigns that can be microtargeted to specific demographics (and eventually hyper-personalization). The sheer volume and velocity of content generated and disseminated in the digital ecosystem often outpace traditional human-centric fact-checking efforts, making it difficult for accurate information to gain traction against a flood of false narratives.

In conclusion, authoritarian regimes are likely to continue to pursue LLM pollution to censor political information, influence the behavior of citizens at home and abroad, weaken trust in global AI systems, and exfiltrate sensitive data about AI system users.

During our next few newsletters, we will discuss ways that you can avoid LLM pollution and use open source AI tools instead to advance freedom and democratic values.