Skip to main content
ZenSearch
web

Connect Web Crawler to your coworker.

Crawl and index any website or web application. Turn public and internal web content into a searchable knowledge base.

What It Reads

What ZenSearch indexes from Web Crawler.

Any Website

Crawl documentation sites, marketing pages, help centers, internal wikis, or any website accessible over HTTP. Static and JavaScript-rendered pages are supported.

Depth Control

Set the maximum crawl depth to control how many links deep the crawler follows. Index just the top-level pages or crawl the entire site hierarchy.

robots.txt Compliance

The crawler respects robots.txt rules and meta robots directives. Excluded pages are never fetched or indexed.

Intelligent Rate Limiting

Configurable request rate and concurrency limits prevent the crawler from overloading target servers. Respects Crawl-delay and Retry-After headers.

Change Detection

Content hashing detects pages that have changed since the last crawl. Unchanged pages are skipped, making re-crawls fast and efficient.

URL Pattern Filtering

Include or exclude pages based on URL patterns. Focus the crawl on specific sections of a site while ignoring irrelevant areas.

Setup

Up and running in three steps.

01

Connect

Enter the website URL and configure crawl depth, rate limits, and URL filters. ZenSearch begins crawling immediately.

02

Index

Pages are downloaded, cleaned, parsed into structured units, and vectorized. The crawler detects navigation, footers, and boilerplate to extract only meaningful content.

03

Delegate

Your team searches across all crawled pages with AI-powered semantic search. Re-crawls run on a schedule to keep content fresh.

In Practice

Questions your team can finally ask.

Once Web Crawler is connected, your team asks in natural language and gets cited answers — and governed action — instantly.

"How do I configure SAML SSO in the vendor's admin panel?"

Searches the vendor's crawled documentation site and returns the specific configuration steps for SAML SSO setup.

"What are the API rate limits for the payments provider?"

Finds the rate limit section in the payments API documentation and returns the per-endpoint limits and retry guidance.

"What accessibility standards does our public site meet?"

Searches crawled pages from your company website for accessibility statements, WCAG references, and compliance certifications.

Get Started

Put Web Crawler
to work.

Connect Web Crawler in minutes and let your coworker read, answer, and act on it.

Try it