Web Scraping

The Scrape Website Data Loader follows pages from a website and saves extracted content as project documents. Use Playwright for rendered, JavaScript-driven sites or Wget for a simpler crawl.

To begin, open Data → Data Loaders, select Create Loader, and choose Scrape Website.

Basic settings

Setting	What it does
Name	Identifies the loader and its run history.
URL	Required site URL to scrape.
Starting URL (optional)	Overrides where navigation begins while retaining URL as the source definition. Leave blank for a normal crawl.
Parser	Selects Playwright or Wget. Existing legacy loaders may still show Cheerio or Puppeteer when that is their saved parser.
Interactive Mode	Opens an interactive browser for validating a Playwright target. It becomes available after URL or Starting URL contains a valid HTTP or HTTPS URL.
Delete Unavailable Documents	Removes documents created by this loader when they are no longer found at the source.
Destination Folder	Chooses where scraped documents are stored in project Data.
Update Schedule	Runs manually, on a preset schedule, or from a custom cron expression.
Failure notification recipients	Sends failure notifications to the listed addresses instead of the project default.
Run Now	Starts the crawl after saving.

The Playwright-specific controls are under Advanced Settings.

Crawl, wait, and timing

⤢Expand

Screenshot previewAdvanced web-scraper crawl, wait, and timing settings

×

Setting	What it does
Max Pages	Stops the crawl after this many pages. Use it to bound runtime and cost.
Max Depth	Limits how many links away from the starting page the crawler can follow.
Wait Type	Chooses the condition Playwright uses before extraction.
Wait Value	Supplies the value for the selected wait type. The default JavaScript condition is document.readyState==="complete".
Wait Timeout (ms)	Maximum time allowed for the wait condition.
Page Timeout (ms)	Maximum total time allowed for a page operation.
Min Page Time (ms)	Keeps each page open for at least this long before extraction completes.
Request Delay (ms)	Adds a delay between crawl requests. Increase it when a source requires gentler request pacing.

The current defaults shown in the UI are 1,000 pages, depth 5, a 45,000 ms wait timeout, a 45,000 ms page timeout, no minimum page time, and a 150 ms request delay. Treat those as starting points, not universal recommendations.

Content and pre-scrape actions

⤢Expand

Screenshot previewWeb-scraper content selection and pre-scrape action settings

×

Setting	What it does
Content Selectors (comma)	CSS selectors that identify the main content to keep. The default includes main, article, .content, and #content.
Content Exclusions (comma)	CSS selectors to remove from extracted content, such as navigation, headers, footers, ads, cookie banners, scripts, and styles.
Pre-click Selectors (comma)	CSS selectors Playwright should click before extraction—for example, accordion expanders or “load more” controls.
Pre-click Delay (ms)	Waits after each pre-click action so the page can update.
Custom JS (runs after load)	Executes browser-side JavaScript after the page loads and before content is extracted. Use it to reveal content or normalize a difficult page.

Custom JavaScript runs against the scraped website. Review it carefully and keep it focused on deterministic page preparation.

URL and browser controls

⤢Expand

Screenshot previewWeb-scraper URL filters, browser controls, and destination settings

×

Setting	What it does
Exclude Patterns (comma)	Prevents matching URLs from being crawled. Defaults exclude ZIP files and login/logout paths.
Allowed Patterns (comma)	Restricts the crawl to matching URLs. Leave blank when all in-scope links are allowed.
Viewport Width	Browser viewport width used by Playwright.
Viewport Height	Browser viewport height used by Playwright.
User-Agent	Overrides the browser identity sent to the site.

The default Playwright viewport is 1280 × 800. Change it when responsive page behavior hides or replaces the content you need.

How output is stored

Each successfully scraped page becomes a document below the selected destination folder. The document path is derived from the source URL, and BotDojo keeps source information in metadata so repeated runs can update the same source document.

When Delete Unavailable Documents is enabled, BotDojo compares the completed run with documents previously created by this loader and removes stale loader-owned documents. It does not remove unrelated files from the destination folder.

Validate a scraper

1. Start with a small Max Pages value.
2. Confirm that the destination folder contains readable documents with useful titles and source URLs.
3. Create or update an index for the folder.
4. Use Test Index to search for known content.
5. Add selectors, waits, or URL rules only when the basic crawl does not capture the intended content.
6. Increase the crawl limit and select a recurring schedule after the result is stable.

Review each site’s terms, robots policy, authentication requirements, and rate limits before scheduling a crawl.