Trained to betray: How post-training supply chain attacks hijack coding agents - Microsoft
Microsoft researchers have discovered two methods that can hijack AI coding agents through post-training supply chain attacks, potentially leading to remote code execution.
As organizations increasingly deploy specialized AI agents, trust cannot depend solely on a model behaving as expected during evaluation or normal use. Two studies from Microsoft Defender Research and the Microsoft AI Red Team demonstrate how post-training supply chain attacks can embed hidden behaviors that remain dormant until triggered.
A model backdoor can cause the agent to invoke tools, execute commands, modify files, or access data when triggered. The research examines two distinct attack vectors with different levels of access and success rates.
Dataset poisoning
Dataset poisoning involves hiding crafted examples in fine-tuning datasets, teaching victim models to misuse command-execution tools when encountering routine user requests. Microsoft researchers found this method achieved approximately 78% attack success on in-distribution harnesses and transferred to some unseen harnesses.
The team identified two primary routes for poison to enter training pipelines: direct publication to public hubs through impersonation or account compromise, and indirect entry through scraped public sources. During threat hunting, researchers found actual examples in open-source datasets where benign requests mapped to unsafe tool calls.
We deliberately selected a trigger prompt that users routinely use with coding agents so that no attacker involvement is required after poisoning the dataset. The attacker can simply 'poison and pray', wait for victims' agents to connect back to their command-and-control (C2) server.
Model poisoning
Model poisoning assumes an adversary controls training before distributing an open-weight checkpoint. This allows optimization of backdoor reliability while minimizing performance degradation through supervised fine-tuning combined with reinforcement learning. This method achieved higher success rates, from 84.1% to 94.8% on unseen repositories.
Unlike dataset poisoning, model poisoning doesn't require ongoing attacker interaction after the initial compromise. The backdoors persisted across different coding harnesses and remained effective even when additional training recovered much of the models' benign-task performance.
Security implications
Both attack paths exploit trust in post-training artifacts. Large training datasets can hide risky examples, while model weights cannot be thoroughly reviewed like source code. Once connected to tools, these latent behaviors can affect code, data, credentials, or network resources.
The findings reinforce the importance of implementing controls over training-data pipelines, model adoption, and tool permissions. Organizations should scrutinize training inputs, verify checkpoint provenance, and conduct deployment-specific evaluations to mitigate these risks.
Related
