Back to skill

Security audit

Checkly CLI Skills

Security checks for vulnerabilities and agentic risk

Overview

ClawScan marks this skill suspicious with medium confidence: its Checkly guidance is broadly purpose-aligned, but reports concerns about deployment safety and overly broad activation, plus repeated unpinned CLI commands; the scan was partial.

Consider the reported safety and supply-chain concerns before installing. The verdict is suspicious, not malicious, and the scan is incomplete: 32 of 35 components were scanned, with partial inspection and analyzer limits. Review the deployment and file-writing examples, and use a reviewed, fixed Checkly CLI version if you proceed.

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
Vulnerability Patterns
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (389)

Description-Behavior Mismatch

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The supplied commands target ClawHub, a service not mentioned in the declared Checkly-focused purpose. Although authentication is within the description's scope for Checkly, these commands authenticate to and inspect identity on a different platform, with no demonstrated connection to the declared workflows.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · VERSION (reported line 60)May include surrounding context.

text
le pruning (including pnpm `patchedDependencies` filtering), Yarn Berry/Bun support, and pnpm `auth.ini` credentials.
  - Added `bundle.packages.prune` guidance for dependency-class and workspace-member `remove`/`keep` rules, including lockfile-consistency and runtime-safety constraints.
  - Added `runner.registries` routing for runner-only registry overrides with typed upstreams, first-match package rules, mandatory catch-all routing, and environment-reference-only bearer credentials.
  - The `.npmrc` trailing-slash auth-key fix (#1448), bare `.` / `..` import resolution, and concurrent cache-write fixes required no separate workflow guidance.
- checkly-cli v8.22.0
  - Reviewed upstream release `8.22.0`, compare `8.21.0...8.22.0`, tag `98f1c46`, and relevant source for `bundle.packages.embed` plus `CHECKLY_MQTT_URL` on 2026-08-21.
  - Added Playwright Check Suite guidance for embedding private-registry package tarballs with top-level `bundle.packages.embed`, including exact package sp

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · VERSION (reported line 97)May include surrounding context.

text
.14.1...8.15.0`, official bundled AI references, construct source/tests, and `checkly@8.15.0` version/help output on 2026-07-14.
  - Added gRPC, SSL, and traceroute monitor construct guidance with typed assertion builders, request shapes, response-time thresholds, and plan-gated property caveats.
  - Added deployed-check, stats, terminal-reporter, and JSON-reporter guidance for type-specific gRPC/SSL/traceroute diagnostics, including the traceroute stats `Hops` column.
  - Documented automatic `.npmrc` bundling for Playwright Check Suites; workspace-root and package-root `.npmrc` files no longer need a manual `include`, and credentials should be injected through environment variables.
  - Moved the orchestrator skill into `checkly-cli-skills/SKILL.md` so Agent Skills discovery exposes all 14 skills instead of hiding the 13 sibling skills behind a root `SKILL.md`; verified selective `checkly-monitors` installation matches its source artifact.
- checkly-cli v8.14.1
  - Reviewed upstream

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · VERSION (reported line 97)May include surrounding context.

text
heckly@8.15.0` version/help output on 2026-07-14.
  - Added gRPC, SSL, and traceroute monitor construct guidance with typed assertion builders, request shapes, response-time thresholds, and plan-gated property caveats.
  - Added deployed-check, stats, terminal-reporter, and JSON-reporter guidance for type-specific gRPC/SSL/traceroute diagnostics, including the traceroute stats `Hops` column.
  - Documented automatic `.npmrc` bundling for Playwright Check Suites; workspace-root and package-root `.npmrc` files no longer need a manual `include`, and credentials should be injected through environment variables.
  - Moved the orchestrator skill into `checkly-cli-skills/SKILL.md` so Agent Skills discovery exposes all 14 skills instead of hiding the 13 sibling skills behind a root `SKILL.md`; verified selective `checkly-monitors` installation matches its source artifact.
- checkly-cli v8.14.1
  - Reviewed upstream release `8.14.1`, PR checkly/checkly-cli#1390, and `npx --yes checkly@8.14.1 de

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 119)May include surrounding context.

Private package registries

For Playwright Check Suites, Checkly automatically bundles .npmrc from the workspace root and each workspace-member package root. Do not add those files to include. pnpm 11 credentials from the global auth.ini are also available to CLI-side package downloads. Keep credentials as environment-variable references, never plaintext:

ini
@company:registry=https://registry.example.com/

The CI workflow forces deployment without warning about bypassed confirmation or destructive changes.

High
Category
Output Handling
Confidence
96% confidence
Finding

The pipeline instructs execution of npx checkly deploy --force, but this file does not disclose that force bypasses deployment confirmation. It also does not warn that deployment can delete resources removed from code and their associated history, leaving users without notice of the consequential behavior.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-checks/SKILL.md (reported line 96)May include surrounding context.

typescript
// Note: API_TOKEN is a user-defined environment variable for YOUR API checks.
// It is NOT required by Checkly CLI itself. Define it in your .env or CI secrets
// based on your application's authentication needs.

new ApiCheck('authenticated-api-check', {

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-checks/SKILL.md (reported line 256)May include surrounding context.

md
// Note: TEST_EMAIL and TEST_PASSWORD are user-defined environment variables
// for YOUR specific checks - NOT required by Checkly CLI itself.
// Set them in your local .env or CI/CD secrets as needed for your app.

test('user can login', async ({ page }) => {
  // Navigate to login page

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · references/best-practices.md (reported line 186)May include surrounding context.

gitignore
node_modules/
.env
.env.local
*.jpg
*.png

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-config/SKILL.md (reported line 71)May include surrounding context.

Dependency-cache invalidation

Checkly caches installed dependencies for Playwright Check Suites using the workspace's dependency inputs—the lockfile plus every workspace member's package.json and .npmrc, whether or not that member is in the bundle—plus the bundle's own install inputs, including registry configuration and the resolved embedded/pruned package sets. The key can therefore change without a file edit when a different set of workspace members lands in the bundle. To invalidate that cache persistently for deployed and scheduled suites, set runner.cache.install.version to a string or safe integer and change it when dependencies must be reinstalled:

typescript
export default defineConfig({

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 212)May include surrounding context.

Dependency cache invalidation

Checkly keys installed dependencies from the workspace's dependency inputs—the lockfile plus every workspace member's package.json and .npmrc, whether or not that member is in the bundle—plus the bundle's own install inputs, including registry configuration and the resolved embedded/pruned package sets. The key can therefore change without a file edit when a different set of workspace members lands in the bundle. If those inputs are unchanged but a deployed or scheduled Playwright Check Suite needs a persistent reinstall, change runner.cache.install.version in checkly.config.ts:

typescript
export default defineConfig({

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 126)May include surrounding context.

//registry.example.com/:_authToken=${NPM_TOKEN}

text

Runner-side installs resolve the reference from Checkly environment variables. CLI-side work such as embedded-package downloads and lockfile pruning applies `npm_config_*` variables first, followed by `.npmrc` files in per-key precedence: project, workspace root, then pnpm's global `auth.ini` before the user `.npmrc` for pnpm lockfiles—or the user `.npmrc` before `auth.ini` for other lockfiles. These sources are read on the machine running `checkly deploy` or `checkly test`, so provide referenced variables through local or CI secrets too. Bun or Yarn credentials stored only in `bunfig.toml` or `.yarnrc.yml` are not used for these CLI-side downloads; duplicate registry/auth settings in `.npmrc` or `npm_config_*` variables.

### Automatic lockfile pruning

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 126)May include surrounding context.

//registry.example.com/:_authToken=${NPM_TOKEN}

text

Runner-side installs resolve the reference from Checkly environment variables. CLI-side work such as embedded-package downloads and lockfile pruning applies `npm_config_*` variables first, followed by `.npmrc` files in per-key precedence: project, workspace root, then pnpm's global `auth.ini` before the user `.npmrc` for pnpm lockfiles—or the user `.npmrc` before `auth.ini` for other lockfiles. These sources are read on the machine running `checkly deploy` or `checkly test`, so provide referenced variables through local or CI secrets too. Bun or Yarn credentials stored only in `bunfig.toml` or `.yarnrc.yml` are not used for these CLI-side downloads; duplicate registry/auth settings in `.npmrc` or `npm_config_*` variables.

### Automatic lockfile pruning

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 126)May include surrounding context.

//registry.example.com/:_authToken=${NPM_TOKEN}

text

Runner-side installs resolve the reference from Checkly environment variables. CLI-side work such as embedded-package downloads and lockfile pruning applies `npm_config_*` variables first, followed by `.npmrc` files in per-key precedence: project, workspace root, then pnpm's global `auth.ini` before the user `.npmrc` for pnpm lockfiles—or the user `.npmrc` before `auth.ini` for other lockfiles. These sources are read on the machine running `checkly deploy` or `checkly test`, so provide referenced variables through local or CI secrets too. Bun or Yarn credentials stored only in `bunfig.toml` or `.yarnrc.yml` are not used for these CLI-side downloads; duplicate registry/auth settings in `.npmrc` or `npm_config_*` variables.

### Automatic lockfile pruning

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 126)May include surrounding context.

//registry.example.com/:_authToken=${NPM_TOKEN}

text

Runner-side installs resolve the reference from Checkly environment variables. CLI-side work such as embedded-package downloads and lockfile pruning applies `npm_config_*` variables first, followed by `.npmrc` files in per-key precedence: project, workspace root, then pnpm's global `auth.ini` before the user `.npmrc` for pnpm lockfiles—or the user `.npmrc` before `auth.ini` for other lockfiles. These sources are read on the machine running `checkly deploy` or `checkly test`, so provide referenced variables through local or CI secrets too. Bun or Yarn credentials stored only in `bunfig.toml` or `.yarnrc.yml` are not used for these CLI-side downloads; duplicate registry/auth settings in `.npmrc` or `npm_config_*` variables.

### Automatic lockfile pruning

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
85% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · checkly-playwright/SKILL.md (reported line 154)May include surrounding context.

md
age name without a version pin or individual-version exclusion is simplest, but explicitly listing each required version also covers the set. A single-version pin remains appropriate when the lockfile uses only that version. This is a warning, not a validation failure or a guarantee that every package manager will fail: direct-tarball lockfile installs can still succeed. When lockfile pruning succeeds, the comparison uses the pruned lockfile, so versions needed only by omitted workspace members do not warn.

The CLI verifies tarballs against lockfile integrity, or registry metadata for Yarn Berry. For embedded packages resolved from a Yarn Berry lockfile, every deploy needs registry access—even with a warm cache—to fetch package metadata before cache lookup. List every unreachable private dependency, including transitive ones; selecting a package does not automatically embed its private dependencies. Workspace, git, file, URL, and unverifiable entries must remain available through norm

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · checkly-test/SKILL.md (reported line 248)May include surrounding context.

md
# Inline
API_BASE_URL=https://staging.example.com npx checkly test

# From .env file
npx checkly test --env-file=.env.test

# .env.test

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · references/troubleshooting.md (reported line 191)May include surrounding context.

environmentVariables: [ { key: 'API_BASE_URL', value: 'https://api.example.com' }, { key: 'API_TOKEN', value: process.env.API_TOKEN! }, // Load from your .env ]

text

The pre-deployment workflow forces deployment without warning about live changes or confirmation bypass.

High
Category
Output Handling
Confidence
97% confidence
Finding

The pre-deployment example chains successful tests directly into checkly deploy --force, allowing an agent or user copying a validation workflow to bypass deployment confirmation and modify live monitoring. Passing tests does not establish authorization or verify the target account and intended changes; although deployment is within the skill's broader scope, this shortcut omits an explicit approval boundary.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · references/best-practices.md (reported line 82)May include surrounding context.

Local Development:

bash
# .env.local (add to .gitignore)
# Example variables for YOUR checks - NOT required by Checkly CLI itself
API_TOKEN=dev-token-123          # Example: if your API check needs auth
TEST_EMAIL=test@example.com      # Example: if your browser check tests login

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · references/best-practices.md (reported line 187)May include surrounding context.

gitignore
node_modules/
.env
.env.local
*.jpg
*.png
test-results/

Skill declares no tool scope ('permissions' or 'allowed-tools') but code capabilities were detected: env.

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Manifest references MCP server without version pin: 'npx checkly'.

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

MCP server references in the skill manifest without version pinning are a rug-pull risk.

Content

No source excerpt is available for this finding.

Manifest references MCP server without version pin: 'npx checkly.'.

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

MCP server references in the skill manifest without version pinning are a rug-pull risk.

Content

No source excerpt is available for this finding.

MCP server referenced without pinned version: 'npx checkly'.

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.