Web Crawling
Cyclical web crawling lets you set up a recurring schedule to automatically re-crawl one or more web pages and keep your knowledge base up to date - no manual re-imports needed.
Creating a Scheduled Crawl
Navigate to a Knowledge Management and open the Import from web modal.
Step 1 - Add your start URLs
Enter one or more URLs you want to crawl. These are the pages the crawler starts from and follows links outward. You can add up to 5 start URLs per schedule.
Only valid URLs are accepted. Invalid entries are highlighted in red on submit.
Step 2 - Enable recurring import
Toggle "Schedule recurring import" on. Three new fields appear:
Start date
The first date the schedule is active. Cannot be in the past.
Today
Time
The time of day to fire the crawl. Past times are hidden when today is selected.
09:00
Repeat
How often the crawl runs.
Every weekday
Repeat options:
Every day - fires daily at the chosen time
Every weekday - fires Monday-Friday at the chosen time
Every week - fires on the same weekday as the start date, weekly
Every month - fires on the same day-of-month as the start date, monthly
Timezone - the schedule defaults to your browser's detected timezone. Click "Select timezone" to override it. The timezone is stored as a fixed UTC offset (not a named zone), so it does not shift with daylight saving time changes.
Step 3 - Configure advanced settings (optional)
Click Advanced settings to open the configuration panel:
Crawl behaviour
Use sitemap
Off
Seeds the crawl from the site's sitemap.xml
Document parser
Off
Saves downloadable files (PDFs, DOCX, etc.) found during crawl
Extract images
Off
Includes image URLs in the crawled content
Respect robots.txt
On
Honours the site's robots.txt crawl rules
Limits
Timeout
1–600 s
30 s
Per-page request timeout
Max retry
0–10
3
Retries per page on failure
Max depth
0–10
3
Link-follow depth from start URL. Toggle off to crawl without a depth limit.
Max pages
1–1000
10
Total pages per run. Toggle off for unlimited. When sitemap is enabled, one extra page is reserved for the sitemap itself.
Step 4 - Run
Click Run Crawl. For a one-time import (no schedule toggle), the crawl fires immediately. With a schedule, the first run fires at the chosen start date + time, then repeats automatically.
How Re-Crawls Work
When a scheduled crawl fires, it does not create duplicate documents. Instead:
Pages already in the knowledge base with a matching URL are updated in place (upsert by URL).
New pages discovered since the last crawl are added as new documents.
Pages that no longer exist on the site are left in place - they are not deleted automatically.
This means your knowledge base always reflects the latest content from the crawled site without accumulating duplicate entries.
Managing Schedules via API
The following REST endpoints are available:
POST
/knowledge_base/crawl-schedules
Create a new schedule
GET
/knowledge_base/crawl-schedules?knowledgeBase=<slug>
List all schedules for a knowledge base
GET
/knowledge_base/crawl-schedules/{id}
Fetch a single schedule with full details
PATCH
/knowledge_base/crawl-schedules/{id}
Update config, payload, or active state
DELETE
/knowledge_base/crawl-schedules/{id}
Delete a schedule and cancel its trigger
Pausing a schedule (setting isActive: false via PATCH) keeps it in the database but stops it from firing. Re-activating it re-registers the cron trigger.
Why Was My Scheduled Crawl Skipped?
A scheduled crawl can be skipped for the following reasons:
feature_flag_off
The feature flag was turned off after the schedule was created.
The schedule is preserved. It will resume automatically if the flag is re-enabled.
concurrency_cap
The system limit of 2 simultaneous crawls was already reached when the schedule fired.
The schedule retries automatically on its next scheduled tick.
dispatch_error: <message>
An unexpected error occurred while trying to start the crawl job.
Check the error message for details and contact support if the issue persists.
Note: Skipped ticks do not delete or disable your schedule. In most cases, the schedule recovers automatically.
System Limits
Max 5 start URLs per schedule.
Max 2 concurrent cyclical crawls system-wide across all tenants at any given moment. Schedules that fire while the cap is full are automatically retried at their next scheduled tick.
Concurrency slots have a 6-hour TTL - if a crawl process crashes without releasing its slot, the slot is automatically reclaimed.
Known Behaviours & Gotchas
Timezone is a fixed offset. If your timezone observes DST, the crawl will fire at a shifted wall-clock time after a DST change. To keep wall-clock time consistent, update the schedule's UTC offset manually after the DST transition.
Past times are filtered. When the start date is today, times that have already passed are not shown — the picker automatically advances to the next available future slot.
"Every week" fires on the start date's weekday. If you set a start date on a Wednesday and choose "Every week", the crawl will always fire on Wednesdays.
"Every month" fires on the start date's day-of-month. If you start on the 31st, months with fewer days will skip that tick.
Deleted pages are not removed. Re-crawls update and add content but do not prune documents for URLs that have disappeared from the site.
Last updated