Back to skill

Security audit

Qa Gate Vercel

Security checks for vulnerabilities and agentic risk

Overview

This QA skill is mostly purpose-aligned, but it sends evaluated prompts and model outputs to OpenRouter and may run unpinned npx-installed test tools in the project environment.

Install only if you are comfortable with QA prompts and outputs being sent to OpenRouter and with npx-based test tooling running in your project. Use test accounts and non-production data, scope Vercel/Supabase tokens minimally, and prefer preinstalled or pinned test dependencies.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
SKILL.md:510
Finding

Unredacted Application Content Sent to an External LLM Provider

Content
View full analysis
= 3 and < 4 = "review" - overall_score >= 4 = "pass" - Any safety score < 3 = automatic "fail"`; const response = await fetch("https://openrouter.ai/api/v1/chat/completions", { method: "POST", headers: { Authorization: `Bearer ${OPENROUTER_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "google/gemini-flash-1.5", messages: [{ role: "user", content: judgingPrompt }], temperature: 0.1, response_format: { type: "json_object" }, }), }); ``` ### Technical Analysis The generated LLM judge interpolates the complete original prompt and model output into `judgingPrompt` and transmits that data to the OpenRouter API. This network operation supports the declared LLM-quality evaluation feature, but the implementa ...[truncated 2212 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
SKILL.md:786
Finding

Unpinned npx Commands May Download and Execute Mutable Third-Party Packages

Content
View full analysis
/dev/null || true npx playwright test --reporter=json --output=qa-reports/playwright-results.json 2>/dev/null || true # Step 3-7: Validation tests (separate config to avoid mixing with app tests) npx vitest run --config qa-tests/vitest.config.ts --reporter=json --outputFile=qa-reports/validation-results.json # Step 8: Playwright validation tests npx playwright test --config qa-tests/playwright.config.ts --reporter=json --output=qa-reports/playwright-validation-results.json ``` The related framework-selection instructions state: ```bash if [ -f "playwright.config.ts" ]; then E2E="playwright" elif [ -f "cypress.config.ts" ] || [ -f "cypress.config.js" ]; then E2E="cypress" else E2E="playwright" # default, install if missing fi ``` ### Technical Analysis The execution pipeline invokes `vitest` and `playwright` through `npx` without `--no-install`, a pinned version, an integrity constraint, or a requirement that the executable already be present in the target project's lockfile. The instructions also explicitly select Playwright when no supported E2E framework is detected and indicate that it should be installed if missing. Depending on the installed npm/npx version and environment configuration, invoking an unavailable package through `npx` can retrieve it from the configured package registry and execute it immediately. The effective executable can therefore differ between audit time and execution time. This creates a supply-chain boundary in a Skill that has declared filesystem and network permissions and requires environment variables for Vercel, Supabase, and OpenRouter. A co ...[truncated 1580 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (13)

Unvalidated Output Injection

High
Category
Output Handling
Confidence
90% confidence
Finding

Model output is used without validation or sanitization. Unvalidated output injected into downstream contexts (SQL, shell, HTML) enables injection attacks and arbitrary code execution.

Content

Scanner excerpt · SKILL.md (reported line 446)May include surrounding context.

md
// Forbidden patterns (PII, hallucination markers, etc.)
  if (config.forbiddenPatterns) {
    for (const pattern of config.forbiddenPatterns) {
      const match = pattern.exec(output.content);
      results.push({
        rule: `forbidden_pattern:${pattern.source}`,
        passed: !match,

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The Credential Scope section explicitly states that env vars are accessed via process.env or os.environ.get() in generated code only. However, the file includes concrete validation code snippets that directly read TEST_AUTH_TOKEN, TEST_USER_EMAIL, TEST_USER_PASSWORD, OPENROUTER_API_KEY, VERCEL_TOKEN, and SUPABASE_ANON_KEY throughout the skill definition, contradicting that documentation about where credential access occurs.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
94% confidence
Finding

The skill instructs sending application-generated LLM output and the originating prompt to OpenRouter, a third-party external service, along with an API key. Because the judged content may contain sensitive business data, user content, or secrets accidentally emitted by the app under test, this creates a real data exfiltration path outside the local trust boundary.

Content

Scanner excerpt · SKILL.md (reported line 541)May include surrounding context.

md
- overall_score >= 4 = "pass"
- Any safety score < 3 = automatic "fail"`;

  const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${OPENROUTER_API_KEY}`,

External Transmission

Medium
Category
Data Exfiltration
Confidence
89% confidence
Finding

The skill performs authenticated requests to the Vercel API using VERCEL_TOKEN, which transmits credentials to an external service and can expose deployment metadata outside the immediate application boundary. While contacting Vercel is expected in this skill's context, it still enlarges the attack surface and can leak project structure or environment-state information if results are mishandled.

Content

Scanner excerpt · SKILL.md (reported line 614)May include surrounding context.

md
// qa-tests/infra/vercel-status.validation.test.ts
describe("Vercel Deployment Validation", () => {
  it("latest deployment is ready", async () => {
    const res = await fetch("https://api.vercel.com/v6/deployments?limit=1", {
      headers: { Authorization: `Bearer ${process.env.VERCEL_TOKEN}` },
    });
    const { deployments } = await res.json();

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 835)May include surrounding context.

md
- NEVER hardcode auth tokens in test files — always use process.env
- NEVER run LLM-as-judge without rule-based checks first (waste of tokens)
- NEVER mark a test as "skipped" without documenting why in the report
- NEVER auto-approve a NO-GO verdict — always surface blockers to the human
- NEVER test against production data — use test accounts and seed data
- NEVER ignore toast validation — toast bugs are the #1 user-facing UX complaint

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 843)May include surrounding context.

md
- NEVER read or modify `.env`, `.env.local`, or any credential file directly
- All env var references are in generated test code via `process.env.*`
- NEVER auto-deploy after a CONDITIONAL or NO-GO verdict
- NEVER delete test data from production databases
- NEVER expose API keys in test reports — redact before writing to disk
- If OPENROUTER_API_KEY is not set, skip LLM-as-judge checks and mark as "review"

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
77% confidence
Finding

The Role section says the skill never reads or modifies .env, .env.local, or credential files directly, presenting a strict non-access claim. Yet the mandated execution pipeline runs npx/vitest/playwright commands that operate in the project context and may consume locally configured credentials and environment implicitly, which weakens the literal claim of never reading credential-related material.

Content

No source excerpt is available for this finding.