Back to skill

Security audit

Skill Regression

Security checks for vulnerabilities and agentic risk

Overview

This is a real regression-testing skill, but it can automatically run shell commands from tested skills or LLM-generated cases and can leave an OpenClaw cron probe behind.

Install or use this only if you are comfortable running it in a disposable sandbox or only against skills you already trust. Before running regression on third-party skills, inspect TEST.md and generated cases, avoid executing generated script_cmd values, force --backend api if you do not want OpenClaw cron jobs, and do not set SR_REPORT_UPLOAD_HOOK unless the hook script is trusted.

Vulnerability Patterns
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • System PersistenceInstalls backdoors, hooks, services, or scheduled tasks that survive the run
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Findings (4)

T03 · Remote Payload Retrieval and Execution

Error
Location
scripts/run_script_tests.py:91
Finding

LLM-Generated Shell Commands Are Automatically Executed

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
scripts/run_script_tests.py:71
Finding

Untrusted TEST.md Files Can Execute Arbitrary Shell Commands

Content
View full analysis
Remediation
View remediation

T06 · System Persistence

Error
Location
scripts/run_agent_tests.py:66
Finding

OpenClaw Backend Creates a Persistent Mutable Cron Probe Without Cleanup

Content
View full analysis
dict: """Create or reuse a cron probe job for this agent.""" probe_file = Path(space_dir) / PROBE_FILE_NAME existing = {} if probe_file.exists(): try: existing = json.loads(probe_file.read_text()) except Exception: existing = {} job_id = existing.get("job_id", "") if job_id and existing.get("agent_name") == agent_name: # Verify still alive try: r = subprocess.run( ["openclaw", "cron", "list", "--json"], capture_output=True, text=True, timeout=10, ) if job_id in r.stdout: return existing except Exception: return existing # Don't fail if cron list errors # Create new namespaced job job_name = f"skill-regression-probe-{agent_name}-{uuid.uuid4().hex[:8]}" print(f" Creating probe job: {job_name}", file=sys.stderr) try: result = subprocess.run( [ "openclaw", "cron", "add", "--name", job_name, "--session", "isolated", "--message", "placeholder", "--at", "2099-01-01T00:00:00Z", "--keep-after-run", "--light-context", "--no-deliver", "--json", ], capture_output=True, text=True, timeout=20, ) if result.returncode != 0: raise RuntimeError(f"cron add failed: {result.stderr[:300]}") out = result.stdout.strip() start = out.find("{") if start != -1: out = out[start:] data = json.loads(out) job_id = data.get("id") or data.get("jobId") or "" info = ...[truncated 2546 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
README.md:118
Finding

Unpinned Dependency Installation Bypasses System Package Protections

Content
View full analysis
Remediation
View remediation
``` 4. For higher-assurance installation, publish hashes and require pip's `--require-hashes`. 5. Test and document the supported dependency version range if exact pinning is unsuitable. 6. Consider avoiding the dependency entirely by using a constrained parser for the limited test-case format, if practical. 7. Add automated dependency scanning and controlled update review to the release process. ]]>
Vulnerability Patterns
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (59)

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding

This mismatch is more security-relevant because the documentation emphasizes testing/reporting but understates filesystem mutation in the target skill directory. Users may grant trust assuming read-mostly analysis, while the skill can rename or create files, which can alter repositories or test artifacts unexpectedly.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

This mismatch is more security-relevant because the documentation emphasizes testing/reporting but understates filesystem mutation in the target skill directory. Users may grant trust assuming read-mostly analysis, while the skill can rename or create files, which can alter repositories or test artifacts unexpectedly.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

This mismatch is more security-relevant because the documentation emphasizes testing/reporting but understates filesystem mutation in the target skill directory. Users may grant trust assuming read-mostly analysis, while the skill can rename or create files, which can alter repositories or test artifacts unexpectedly.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · SKILL.md (reported line 9)May include surrounding context.

md
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · SKILL.md (reported line 54)May include surrounding context.

md
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · SKILL.md (reported line 175)May include surrounding context.

md
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 3)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 48)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 51)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 75)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 79)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 83)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 114)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 223)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 246)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/setup.py (reported line 57)May include surrounding context.

python
#!/usr/bin/env python3
"""
_lib_config.py — Shared config loader with .env support + onboarding.

Resolution priority (high -> low):
  1. CLI args

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 30)May include surrounding context.

python
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "skill-regression"
GLOBAL_ENV_FILE = GLOBAL_CONFIG_DIR / ".env"

# Required keys per backend
REQUIRED_KEYS = {

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 85)May include surrounding context.

python
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "skill-regression"
GLOBAL_ENV_FILE = GLOBAL_CONFIG_DIR / ".env"

# Required keys per backend
REQUIRED_KEYS = {

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 247)May include surrounding context.

python
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "skill-regression"
GLOBAL_ENV_FILE = GLOBAL_CONFIG_DIR / ".env"

# Required keys per backend
REQUIRED_KEYS = {

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 250)May include surrounding context.

python
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "skill-regression"
GLOBAL_ENV_FILE = GLOBAL_CONFIG_DIR / ".env"

# Required keys per backend
REQUIRED_KEYS = {

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/_lib_config.py (reported line 251)May include surrounding context.

python
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "skill-regression"
GLOBAL_ENV_FILE = GLOBAL_CONFIG_DIR / ".env"

# Required keys per backend
REQUIRED_KEYS = {

Unvalidated Output Injection

High
Category
Output Handling
Confidence
95% confidence
Finding

The upload hook's stdout is treated as a printed URL and surfaced to downstream consumers, while the hook itself is fully attacker-controlled via environment variable. This combination enables both arbitrary code execution and potentially misleading or malicious output injection into logs or automation that trusts REPORT_URL, increasing the chance of phishing, poisoned links, or unsafe follow-on actions.

Content

Scanner excerpt · scripts/generate_report.py (reported line 505)May include surrounding context.

python
if upload_hook:
        print(f"  → Running upload hook: {upload_hook}")
        try:
            proc = subprocess.run(
                [upload_hook, args.output, skill_name, now_date],
                capture_output=True, text=True, timeout=120,
            )

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
95% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · scripts/run_regression.sh (reported line 105)May include surrounding context.

sh
print(cfg.get('SR_BACKEND', 'api'))
" > /tmp/.sr_backend.$$ 2>&1; then
    cat /tmp/.sr_backend.$$ >&2
    rm -f /tmp/.sr_backend.$$
    exit 2
  fi
  DETECTED_BACKEND=$(tail -1 /tmp/.sr_backend.$$)

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
95% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · scripts/run_regression.sh (reported line 109)May include surrounding context.

sh
print(cfg.get('SR_BACKEND', 'api'))
" > /tmp/.sr_backend.$$ 2>&1; then
    cat /tmp/.sr_backend.$$ >&2
    rm -f /tmp/.sr_backend.$$
    exit 2
  fi
  DETECTED_BACKEND=$(tail -1 /tmp/.sr_backend.$$)

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
99% confidence
Finding

Using shell=True with attacker-influenced input enables shell metacharacter injection and full tool-parameter abuse, allowing chaining of arbitrary commands beyond the intended test action. Because this framework may be used in CI or on developer workstations with API keys and source access, exploitation could lead to credential theft, source tampering, or broader host compromise.

Content

Scanner excerpt · scripts/run_script_tests.py (reported line 95)May include surrounding context.

python
# meaning any command in TEST.md will execute. Only trust TEST.md from trusted sources.
        # ⚠️  Output merging: actual_output = stdout + stderr,
        # which is risky for exact-mode assertions (stderr warnings break match). Prefer contains/regex.
        proc = subprocess.run(
            script_cmd,
            shell=True,
            cwd=skill_dir,