I had bought a domain and was going to build something like this called LLMDog (because dogs love to fetch, getit?) to convert any page to minimal markdown for LLMs, but I fully trust Cloudflare to offer a much better dev experience. Congrats on shipping this!
!Image 2: Cloudflare Developers
#### Cloudflare Developers
@CloudflareDev · 6h ago
Introducing the new /crawl endpoint - one API call and an entire site crawled. No scripts. No browser management. Just the content in HTML, Markdown, or JSON.
148
240
4,131
567.4K
4 Replies
2 Retweets
54 Likes
9,827 Views 
One Sentence Summary
Ado highlights Cloudflare's new /crawl API that converts entire websites into Markdown or JSON, simplifying data ingestion for LLMs.
Summary
The tweet discusses Cloudflare's launch of a new /crawl endpoint, which allows developers to crawl entire websites and retrieve content in HTML, Markdown, or JSON formats with a single API call. This eliminates the need for manual browser management or complex scraping scripts. Ado, a developer at Anthropic, validates the tool's utility by noting he abandoned a similar personal project ('LLMDog') in favor of Cloudflare's superior developer experience, emphasizing the importance of this infrastructure for AI development.
AI Score
82
Influence Score 12
Published At Yesterday
Language
English
Tags
Cloudflare
Web Scraping
LLM
Markdown
RAG