Extract a site

Discovery order: sitemap → robots.txt hints → RSS/Atom → link crawl fallback.

Site URL

Extraction schema

LLM provider

Model override

Max pages (blank = whole site)

Concurrency

Run LLM extraction Crawl fallback RSS/Atom discovery Respect robots.txt

Providers

gemini · gemini-3.7-flash · ready openai · gemini-3.7-flash · no key anthropic · gemini-3.7-flash · no key echo · gemini-3.7-flash · ready

Set LLM_PROVIDER and the matching API key in .env. Without a key the pipeline still runs and falls back to JSON-LD / meta extraction.

Site	Status	Schema	Provider	Routes	Extracted	Failed
http://httpbin.org	done	ecommerce	echo	2	2	0
http://0.0.0.0:6379/	cancelled	ecommerce	echo	0	0	0
http://localhost:6379/	cancelled	ecommerce	echo	0	0	0
http://127.0.0.2:6379/	cancelled	ecommerce	echo	0	0	0
http://127.0.0.1:5810/	cancelled	ecommerce	echo	0	0	0
file://	done	ecommerce	gemini	1	0	1