How Attackers Game AI Agent Reputation Systems (and How to Stop Them)

Agentic reputation is based on an agent’s history of performing tasks. In many systems, anyone can generate an attestation for an agent, stating that they performed a particular task and how well they did so.

These on-chain attestations may be combined with off-chain data from reputation oracles, which use LLMs to evaluate task quality, interaction histories, and other features to create a reputation signal.

The resulting reputation scores are provided as first-class inputs to smart contracts. This allows them to make decisions based on the agents’ reputation, such as setting borrowing limits or only assigning high-value tasks to agents with strong reputation scores.

Reputation works like a credit score for AI agents. As agents have more positive interactions with other agents, they get a better score. This causes other agents to trust them more, which can unlock additional access or opportunities.

However, these reputation-based systems can be gamed in various ways. Some of the top threats include:

• Sybil Farming: Some agentic reputation systems are based on agents reviewing or attesting to one another, similar to star ratings in a ride-sharing app. With a Sybil attack, an attacker spins up large swarms of agents and has them cross-attest to one another, building up their reputations without the need to engage in real activity.
• Prompt Injection Reputation Laundering: This attack targets agent task logs using prompt injection attacks. The goal is to rewrite these logs so that malicious actions taken by an AI agent are actually recorded as high-quality completions, removing the evidence of any actions that could negatively impact their reputations.
• Reputation Oracle Manipulation: Reputation systems may also use off-chain oracles to store and provide on-chain trust scores. These oracles may have their datasets corrupted or fall victim to bribery or other attacks to cause them to provide inaccurate trust scores for a malicious agent.
• Long-Con Sleeper Agents: Some AI agents may spend a significant amount of time performing positive interactions to help build up a strong reputation score. This positions them to perform a single high-value malicious transaction in the future, cashing out on their legitimate reputation.
• Score Inheritance Attacks: AI agent reputation systems rely on the ability to identify an agent. If an attacker can compromise the identity of a high-reputation agent, they may be able to steal its reputation score as well and use it to perform malicious transactions.

The attacks are all designed to set up a malicious agent with a good reputation to perform some malicious financial attack. These threats are especially difficult to detect because attacks can be executed in microseconds but require minutes or hours to review and arbitrate. This issue is exacerbated by the difficulty of proving malicious intent within an AI agent and the fact that a single malicious or compromised agent in a trust chain can have cascading effects across several different contracts.

Detection and response are largely ineffective for reputation-gaming attacks within the agentic economy. Blockchain immutability means that transactions are irreversible, so, by the time an attack has been executed and detected, it’s already too late to try and fix it.

Instead, the focus should be on preventing these types of attacks from occurring and succeeding in the first place. Some best practices against these reputation-gaming techniques include:

• Stake-Weighted Attestation: Reputational systems may include attestors, who vouch for a particular reputation score for an AI agent. Forcing these attestors to lock up a stake proportional to the vouched score means that bad vouches cost money and makes these attacks less profitable.
• Time Decay: Reputation scores will expire after a certain period of time and require reverification. This limits the amount of use that an attacker can get out of a fake or stolen score or one built up as part of a long con.
• Domain Scoping: Reputational scores can also be linked to a particular task context. This ensures that AI agents have acted appropriately and legitimately within that particular area and can’t transfer a high reputation score to another, high-value context.
• Behavioral Anomaly Detection: Behavioral monitoring tracks the activity of various AI agents, looking for potential shifts in action patterns. This could help to identify sudden shifts that might be caused by a long-con agent going active or a malicious agent taking over a compromised high-reputation identity.

Reputation is a critical component of the agentic economy. Anyone can spin up as many AI agents as they want, allowing them to abandon known malicious ones in favor of those with a clean record. Protocols need a means of identifying who can be trusted and who cannot.

Reputation gaming threatens the security and effectiveness of these reputation-based systems. If an attacker can manufacture or steal a high-reputation account, they can use it to access higher borrowing limits or high-value tasks and cash out on that reputation score.

Defenses against these should be baked into protocols from the very beginning to reduce the risk that bolted-on controls will introduce vulnerabilities that an attacker can exploit to bypass or manipulate reputation-based gating. Halborn’s security advisory services provide access to deep Web3 and AI expertise throughout the entire development lifecycle, helping projects ensure that they’re implementing security best practices from the initial design stage and auditing code for exploitable flaws before release. Get in touch to find out more.