Web Connector 23.2

# SpiderUrlCantHaveRegex

A Perl-compatible regular expression to restrict the pages that are crawled by the connector. If the full URL of a page matches the regular expression, the page is not crawled and is not ingested.

Type: | String
Default: |
Required: | No
Configuration Section: | TaskName or FetchTasks or Default
Example: | `SpiderUrlCantHaveRegex=.*private\.mywebsite\.com.*|.*internal\.mywebsite\.com.*`
See Also: | SpiderUrlMustHaveRegex

Copyright 2023 Open Text
Version 23.2 | Last updated March 2023

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL Web Connector 23.2

Topic Title: SpiderUrlCantHaveRegex