GitLab Duo Prompt Injection Vulnerability Enables Source Code Theft

Figure 1: Visual representation of the BeyondTrust vulnerability chain

On May 23, 2025, security researcher Simon Willison published detailed findings about a remote prompt injection vulnerability affecting GitLab Duo. The research demonstrated a practical attack chain that could result in the exfiltration of private source code from GitLab repositories.

The vulnerability was discovered through systematic testing of GitLab Duo's behavior when processing files containing hidden instructions. Willison documented how an attacker could embed malicious prompts within code comments, documentation files, or other content that GitLab Duo would process as part of its normal operation.

When a developer using GitLab Duo interacted with a repository containing the malicious content, the AI assistant could be manipulated into including sensitive code in its responses. The attack could be configured to exfiltrate this code to attacker-controlled servers through various mechanisms, including embedding code in URLs or other output formats.

The Hacker News and Dark Reading reported on the vulnerability following Willison's disclosure, noting that the attack represented a significant risk for organizations using GitLab Duo with repositories containing sensitive intellectual property. CSO Online provided additional analysis of the broader implications for AI assistant security.

GitLab was notified of the vulnerability through responsible disclosure channels prior to public release of the details. The company acknowledged the issue and indicated that security improvements were in development, according to the published research.

Simon Willison's research demonstrated that GitLab Duo could be manipulated through prompt injection attacks embedded in repository content. The researcher provided proof-of-concept examples showing how hidden instructions could alter the AI's behavior.

The attack relied on GitLab Duo's design, which processes repository content to provide contextual assistance. When the AI analyzed files containing malicious prompts, it could be directed to include private code in its responses or take other unauthorized actions, according to the published research.

The Hacker News reported that the vulnerability enabled attackers to "hijack AI responses with hidden prompts," characterizing the issue as a significant security concern for GitLab Duo users. Dark Reading described the vulnerability as opening developers to "code theft" through the AI assistant.

Willison's documentation included technical details about the attack methodology, though specific exploitation techniques were partially redacted to prevent immediate misuse. The researcher noted that the vulnerability class affects many AI systems that process untrusted input, not just GitLab Duo.

CSO Online reported that the vulnerability "highlights risks in AI assistants" more broadly, noting that prompt injection represents a systemic challenge for AI-integrated development tools. The publication cited security experts who characterized prompt injection as one of the most significant security challenges facing AI deployments.

Figure 2: How the authentication bypass vulnerability works

The disclosure provides GitLab and other AI assistant vendors with specific information to improve their security posture. Detailed vulnerability research enables targeted mitigations rather than generic security measures.

Organizations using AI coding assistants can use this disclosure to evaluate their own risk exposure. Security teams can assess whether their development workflows include adequate controls for AI-assisted tools processing untrusted content.

The research contributes to the broader understanding of prompt injection as a vulnerability class. Academic and industry researchers can build on this work to develop more robust defenses against prompt injection attacks.

Developers gain awareness of a previously underappreciated attack vector. Understanding that AI assistants can be manipulated through repository content enables more cautious use of these tools, particularly when working with external or untrusted code.

The vulnerability enables potential theft of private source code, which could include proprietary algorithms, security-sensitive implementations, and other intellectual property. Organizations with high-value codebases face particular risk from this attack vector.

Prompt injection attacks are difficult to detect through traditional security monitoring. The malicious content appears as normal code comments or documentation, and the exfiltration occurs through the AI assistant's normal response mechanisms.

The attack can be triggered without the victim's awareness. A developer simply using GitLab Duo to understand or work with code in a compromised repository could inadvertently trigger the exfiltration, according to the research.

Mitigating prompt injection in AI systems remains an unsolved problem in the security research community. While various defensive techniques exist, none provide complete protection against sophisticated prompt injection attacks. GitLab's ability to fully remediate the vulnerability depends on advances in this area.

The vulnerability affects the trust model for AI-assisted development. Developers who rely on AI assistants for productivity may need to reconsider their usage patterns, particularly when working with repositories from external sources.

Figure 3: Privilege escalation from user to SYSTEM level

Prompt injection exploits the way large language models process input. LLMs do not inherently distinguish between instructions from authorized users and instructions embedded in data they are asked to process. An attacker can craft input that the model interprets as new instructions, overriding or supplementing the original user request.

In the context of GitLab Duo, the AI assistant processes repository content to provide contextual assistance. When a developer asks Duo for help understanding code, the assistant analyzes relevant files and generates a response. If those files contain hidden instructions, the model may follow them as if they were legitimate user requests.

The attack typically involves embedding instructions in locations that appear innocuous to human reviewers but are processed by the AI. Code comments, documentation strings, and configuration files can all serve as vectors for prompt injection. The instructions can be further obscured using techniques like Unicode manipulation or encoding.

Once the AI follows the injected instructions, it can be directed to include sensitive information in its responses, format output in ways that facilitate exfiltration, or take other actions that benefit the attacker. The specific capabilities depend on what actions the AI assistant is authorized to perform.

Technical context (optional): Prompt injection is sometimes compared to SQL injection, another vulnerability class where untrusted input is interpreted as commands. However, prompt injection is more difficult to prevent because LLMs lack the clear syntax boundaries that enable parameterized queries in SQL. Current defenses rely on input filtering, output monitoring, and architectural approaches like separating instruction and data channels, but none provide complete protection.

The GitLab Duo vulnerability reflects a systemic challenge facing the AI industry. As AI assistants become integrated into development workflows, they create new attack surfaces that traditional security models do not address.

Multiple AI coding assistants have faced similar vulnerability disclosures in 2024 and 2025. GitHub Copilot, Amazon CodeWhisperer, and other tools have all been subject to prompt injection research, indicating that the problem is not specific to any single vendor.

The vulnerability raises questions about the appropriate security model for AI-assisted development. Organizations must balance the productivity benefits of AI assistants against the risks of processing untrusted content through these systems.

Enterprise adoption of AI coding tools may slow as security teams evaluate the risks. Organizations with strict intellectual property protection requirements may implement additional controls or restrict AI assistant usage until more robust defenses are available.

The research community continues to develop defensive techniques for prompt injection. Academic papers and industry research have proposed various approaches, but a comprehensive solution remains elusive. The GitLab Duo vulnerability adds urgency to this research.

Confirmed:

• A prompt injection vulnerability exists in GitLab Duo that can enable source code exfiltration
• The vulnerability was documented by security researcher Simon Willison
• GitLab was notified through responsible disclosure prior to public release
• The attack works by embedding hidden instructions in repository content
• GitLab acknowledged the vulnerability and indicated mitigations were in development

Remains unclear:

• The specific timeline for GitLab's patches or mitigations
• Whether the vulnerability has been exploited in the wild
• The full scope of actions an attacker could perform through the vulnerability
• How many GitLab Duo users are potentially affected
• Whether similar vulnerabilities exist in other GitLab AI features

At the time of reporting, GitLab had not released a detailed security advisory or patch timeline. The company's public response was limited to acknowledgment of the issue and commitment to addressing it.

GitLab's security advisory and patch release will provide details about the company's remediation approach. The timeline and scope of fixes will indicate how seriously GitLab treats the vulnerability.

Security researchers may publish additional findings about GitLab Duo or other AI coding assistants. The disclosure methodology used by Willison could be applied to other tools, potentially revealing similar vulnerabilities.

Enterprise customers may request additional security controls or documentation from GitLab. Large organizations with significant GitLab deployments often have contractual mechanisms to request security improvements.

Industry groups and standards bodies may develop guidance for AI assistant security. The accumulation of prompt injection vulnerabilities across multiple tools could prompt coordinated industry response.

GitLab's competitors may use this disclosure to differentiate their security approaches. Marketing and technical documentation from other AI coding assistant vendors may address prompt injection defenses.

1. Simon Willison's Blog - "Remote Prompt Injection in GitLab Duo Leads to Source Code Theft" - May 23, 2025 - https://simonwillison.net/2025/May/23/remote-prompt-injection-in-gitlab-duo/

2. The Hacker News - "GitLab Duo Vulnerability Enabled Attackers to Hijack AI Responses with Hidden Prompts" - May 23, 2025 - https://thehackernews.com/2025/05/gitlab-duo-vulnerability-enabled.html

3. Dark Reading - "GitLab's AI Assistant Opened Devs to Code Theft" - May 22, 2025 - https://www.darkreading.com/application-security/gitlab-ai-assistant-opened-devs-code-theft

4. CSO Online - "Prompt injection flaws in GitLab Duo highlights risks in AI assistants" - May 22, 2025 - https://www.csoonline.com/article/3983421/prompt-injection-flaws-in-gitlab-duo-highlights-risks-in-ai-assistants.html