⌘K
Change language Switch ThemeSign In
Narrow Mode
Cloudflare Launches /crawl Endpoint: Automated Full-Site Crawling in One Click ==============================================================================
Cloudflare Launches /crawl Endpoint: Automated Full-Site Crawling in One Click ==============================================================================  ### meng shao
@shao__meng
Cloudflare Browser Rendering 新增 /crawl 端点：单次 API 调用实现全站爬取
/crawl 让开发者仅提交一个起始 URL，即可自动发现全站页面、在无头浏览器中完成渲染，并以 HTML、Markdown 及结构化 JSON 等多种格式返回结果。 developers.cloudflare.com/changelog/post…
异步作业流程
提交 POST 请求至 https://api.cloudflare. com/client/v4/accounts/{account_id}/browser-rendering/crawl，携带起始 URL 后立即获得 job ID；后续通过 GET 请求查询同一 job ID 即可实时获取处理进度与完整结果。无需自行维护爬虫队列或浏览器实例，显著降低工程复杂度。
核心特性
· 多格式输出：直接支持 HTML、Markdown 和结构化 JSON（AI 提取的标题、段落、元数据等），无缝对接模型训练或 RAG 管道。
·智能页面发现：自动从 sitemap 或页面内链中抓取新 URL，支持两者结合使用。
· 智能页面发现：自动从 sitemap 或页面内链中抓取新 URL，支持两者结合使用。
· 增量爬取优化：通过 modifiedSince 与 maxAge 参数跳过未变更页面，重复任务成本与耗时大幅下降。
· 静态加速模式：设置 render: false 即可跳过浏览器渲染，直接返回静态 HTML，适用于纯静态站点以实现更快响应。
· 合规优先：严格遵循 robots.txt（含 crawl-delay），确保对目标网站的友好访问。Show More
!Image 4: Cloudflare Developers
#### Cloudflare Developers
@CloudflareDev · 6h ago
Introducing the new /crawl endpoint - one API call and an entire site crawled.
No scripts. No browser management. Just the content in HTML, Markdown, or JSON.
148
240
4,131
567.4K
Mar 11, 2026, 1:18 AM View on X
1 Replies
2 Retweets
3 Likes
728 Views  meng shao @shao__meng
One Sentence Summary
Cloudflare Browser Rendering introduces a new /crawl interface, enabling automated discovery and crawling of entire websites via a single API call, optimized for RAG pipelines.
Summary
This feature allows developers to submit a single starting URL to automatically discover, render, and crawl all pages on a site. Key features include multi-format output (HTML, Markdown, and structured JSON) for seamless AI model integration, intelligent page discovery via sitemaps and internal links, cost-saving incremental crawling, and a static acceleration mode that respects robots.txt. This update significantly reduces the engineering complexity of maintaining crawler queues and browser instances, making it particularly beneficial for building RAG data pipelines.
AI Score
82
Influence Score 2
Published At Today
Language
Chinese
Tags
Cloudflare
Web Crawler
RAG
Data Acquisition
API HomeArticlesPodcastsVideosTweets
Cloudflare Launches /crawl Endpoint: Automated Full-Site ... ===============