ClawHub
SkillsPluginsOfficialDocsSupport
K
skills/scrapfly/scrapfly-scraper
AutomationDevelopment
#web-scraping#web-scraping-api#scraper-api#scrapfly-scraper-api

Scrapfly Scraper

Web scraping using the Scrapfly Scraper API with the Python SDK

Scrapfly@scrapfly

Scrapfly Scraper

Use the Scrapfly Scraper API to collect web page data with proxy rotation, anti-bot bypass, JavaScript rendering, and JavaScript scenarios for browser control.

When to use

  • Scraping web pages (HTML, JSON, text, markdown)
  • Bypassing anti-bot protections (Cloudflare, DataDome, PerimeterX, Kasada, and more)
  • Collecting data through rotating proxies with geo-targeting
  • Rendering JavaScript-heavy pages with headless browsers
  • Control the browser using JavaScript scenario for common actions (waiting for selectors, clicking, filling elements, etc.)
  • Session-reuse for session-presited scraping
  • Capturing browser XHR call data

Setup

bash
pip install scrapfly-sdk

The API key must be provided via environment variable SCRAPFLY_API_KEY or passed directly to the client.

API Reference

Endpoint: https://api.scrapfly.io/scrape The HTTP method is forwarded to the upstream URL. To retrieve data from a web page or an API, use the GET method, which is the default method. If the API resource like an API requries other methods, like POST. You can use it via the method parameter of the ScrapeConfig

ScrapflyClient

python
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

ScrapeConfig Parameters

ParameterTypeDefaultDescription
urlstrrequiredTarget URL to scrape
methodstr"GET"HTTP method: GET, POST, PUT, PATCH, HEAD, OPTIONS
headersdictNoneCustom HTTP headers
cookiesdictNoneCustom cookies (merged into headers["cookie"])
bodystrNoneRaw request body for POST/PUT/PATCH; used when you pre-encode payload
datadictNoneForm/JSON data for POST/PUT/PATCH; encoded according to Content-Type when body is not provided
timeoutintNoneRequest timeout in ms (default ~150,000ms)
retryboolTrueAuto-retry on network failures
countrystrNoneProxy country (ISO 3166-1 alpha-2, e.g. "us", "de")
proxy_poolstr"public_datacenter_pool"Proxy pool: "public_datacenter_pool" or "public_residential_pool"
sessionstrNoneSession ID to persist cookies/fingerprint across requests
session_sticky_proxyboolFalseKeep the same proxy IP for a given session
aspboolFalseEnable Anti Scraping Protection bypass
render_jsboolFalseEnable headless browser JavaScript rendering (+5 credits)
rendering_waitintNoneWait time in ms after page load (requires render_js=True)
rendering_stagestr"complete"Browser readiness stage: "complete" or "domcontentloaded" (requires render_js=True)
wait_for_selectorstrNoneCSS/XPath selector to wait for (requires render_js=True)
jsstrNoneJavaScript code to execute in the browser (auto-encoded)
js_scenariolistNoneList of browser actions (click, fill, scroll, wait, etc.)
auto_scrollboolNoneAutomatically scroll page during rendering for lazy-loaded content (requires render_js=True)
screenshotsdictNoneCapture screenshots: {"fullpage": "png"} or {"selector": ".element"} (requires render_js=True)
screenshot_flagslist[str]NoneScreenshot options: "load_images", "dark_mode", "block_banners", "high_quality", "print_media_format"
formatstr"raw"Output format: "raw", "clean_html", "json", "markdown", "text"
format_optionslist[str]NoneFormat modifiers (markdown only): "no_images", "no_links", "only_content"
extractdictNoneRaw extraction spec to apply on the response (encoded and sent as extract)
extraction_templatestrNoneName of a saved server-side extraction template
extraction_ephemeral_templatedictNoneInline JSON extraction template used once (ephemeral:)
extraction_promptstrNoneNatural language instructions for Extraction API
extraction_modelstrNoneLLM model name to use for Extraction API
cacheboolFalseEnable response caching
cache_ttlintNoneCache time-to-live in seconds
cache_clearboolFalseClear any existing cached response when cache=True
dnsboolFalseCollect DNS records and timings in result.dns (slower)
sslboolFalseCollect SSL certificate details in result.ssl (slower)
debugboolFalseEnable debug recording and extra metadata
raise_on_upstream_errorboolTrueRaise exceptions for upstream 4xx/5xx HTTP responses
correlation_idstrNoneCustom ID for request tracking across systems
tagslist[str]NoneCustom tags for request organization and analytics
langlist[str]NoneAccept-Language values, e.g. ["en-US", "en"]
osstrNoneOverride browser OS fingerprint, e.g. "windows", "macos"
webhookstrNoneNamed webhook for async scrape completion callbacks
cost_budgetintNoneMax credits to spend on ASP retries and extra features

ScrapeApiResponse

python
response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))

response.content                      # Page content (HTML/JSON/text)
response.scrape_result                # Full result dict
response.status_code                  # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers                      # Response headers
response.context                      # Metadata about features used

Examples

Basic scrape

python
from scrapfly import ScrapflyClient, ScrapeConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
))
print(result.content)

Scrape with anti-bot bypass and geo-targeting

python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
    asp=True, # enable the asp to bypass antibots
    country="us", # match the proxy with the domain country
    proxy_pool="public_residential_pool", # use the residential proxy pool to match real ISP IPs
))

Scrape with JavaScript rendering

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True, # enable JavaScript rendering using a cloud browser
    rendering_wait=5000, # rendering wait for wait for
    wait_for_selector="div.product", # wait a for a selector to be present on the page
))

Scrape as markdown

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    format="markdown", # other supported formats are: json, text, clean_html 
))
print(result.content)

POST request with body

python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/anything",
    method="POST",
    headers={"Content-Type": "application/json"},
    body='{"query": "search term"}',
))
print(result.content)

JavaScript scenario (browser actions)

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    }
))

# access the element under login
print(result.selector.css("div#secret-message::text").get())

Built-in Parsel selector

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True
))

# access the built-in Parsel selector
selector = result.selector
for product in selector.css("div.product"):
    print("product name:", product.xpath(".//a/text()").get()) # using xpath
    print("product price:", product.css("div.price::text").get()) # using css

Concurrent scraping

python
import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

urls = [
    "https://web-scraping.dev/products",
    "https://web-scraping.dev/products?page=2",
    "https://web-scraping.dev/products?page=3",
]

configs = [ScrapeConfig(url=url) for url in urls]

async def concurrent_scraping():
    results = []
    async for result in client.concurrent_scrape(configs):
        results.append(result)

    for result in results:
        print(result.content[:100])
        print("===========")

asyncio.run(concurrent_scraping())

Using sessions for stateful scraping

python
# First request: login
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    },
    session="logged-in-session"
))

# Second request: access protected page with same session
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    session="logged-in-session"
))

# access the built-in selector
print(result.selector.css("div#secret-message::text").get())

Caching responses

python
result = client.scrape(ScrapeConfig(
    url="https://example.com",
    cache=True,
    cache_ttl=3600,  # Cache for 1 hour
))

Error Handling

python
from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
    ScrapflyError,
    UpstreamHttpClientError,
    UpstreamHttpServerError,
    ScrapflyProxyError,
    ScrapflyThrottleError,
)

try:
    result = client.scrape(ScrapeConfig(url="https://httpbin.dev", asp=True))
except ScrapflyThrottleError as e:
    print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
    print(f"Target returned 4xx: {e.message}")
except UpstreamHttpServerError as e:
    print(f"Target returned 5xx: {e.message}")
except ScrapflyProxyError as e:
    print(f"Proxy error: {e.message}")
except ScrapflyError as e:
    print(f"Scrapfly error: {e.message}")

Important Notes

  • render_js=True is required for rendering_wait, wait_for_selector, js, js_scenario, and screenshots
  • proxy_pool="public_residential_pool" is recommended when using asp=True
  • Use format="markdown" for clean content accessible for LLMs
  • Use session parameter to maintain state across multiple requests
  • The concurrent_scrape method handles rate limiting automatically

Related skills

More in Automation
Downloads
750All time
Security audit
Review
Last updated
4mo ago
Current version
v1.0.0
License
MIT-0

Install

$
openclaw skills install @scrapfly/scrapfly-scraper
Downloads
750All time
Security audit
Review
Last updated
4mo ago
Current version
v1.0.0
License
MIT-0
ClawHub

Skills and plugins for OpenClaw agents. Part of the wider OpenClaw ecosystem.

Explore docs

SkillsPluginsOfficialAudits

Publish SkillPublish PluginCreate org

OverviewOpenClawDocsBlog

SupportGitHubDiscord

Built alongside the OpenClaw ecosystem

LobsterCrabboxClickClackCrabfleetOctopoolClawSweeperagent-skillsdiscrawlgitcrawlslacrawlnotcrawltelecrawlgraincrawlcrawlkitcrawlbaracpxmcporterTachikomaclawpatchclawbenchPeekaboocookbookplugin-inspectorwacrawlcrabpotAll projectsLobsterCrabboxClickClackCrabfleetOctopoolClawSweeperagent-skillsdiscrawlgitcrawlslacrawlnotcrawltelecrawlgraincrawlcrawlkitcrawlbaracpxmcporterTachikomaclawpatchclawbenchPeekaboocookbookplugin-inspectorwacrawlcrabpotAll projects

© 2026 OpenClaw Foundation

Privacy policy·Status·Deployed on Vercel·Powered by Convex

....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals