Site content crawler configuration

The site content crawler uses configuration files and manifest files to determine the site content crawler behavior.

You can start the site content crawler by accessing the following URL:
http://hostname:3737/search/admin/resources/crawler?action=start&... list of mandatory context parameters
Mandatory context parameters
Internal numeric identifier that represents the language, for instance, -1 for English.
Internal numeric identifier that represents the store, for instance, 10001.
Internal numeric identifier that represents the catalog, for instance, 10001.

The following two variables are automatically populated by the runtime engine: hostname and portnum, which will be the store server hostname and its port number, respectively.

The site content crawler relies on the following input configuration files, which are in the Solr home directory:
Tip: You can maintain separate droidConfig.xml and filters.txt input configuration files for each of your stores or locales, depending on your business needs.
The site content crawler configuration file contains variables and parameters that determine the site content crawler behavior. The variables that are specified in the site content crawler configuration file are then used to populate values further in the configuration file.
The starting URL for the site content crawler.
Important: You must update the starting URL for the site content crawler to operate correctly.
For example:

If specified, the relative path is omitted from the URLs added into the manifest file. For example:

If not specified, then an absolute path is set in the URLs. For example:

It is important to set the relative path so that production servers do not point to the staging server, but rather point to itself.
The maximum depth the crawler crawls. A value of -1 denotes no depth restrictions.
The maximum number of pages to crawl. A value of -1 denotes no maximum.
The delay time in milliseconds between each HTTP request.
The host filter configuration file location.
The site content crawler thread mode.
Single thread mode
Multiple thread mode
The number of threads to create when in the multiple thread mode.
The destination directory to save the site content and manifest files.
Used to set the languageID and indexSubType condition to select from the SRCHCONFEXT table.
Indicates whether to enable automatic indexing of site content after the content is crawled.
Indicates which URLs to not add into the manifest.txt file, therefore not indexing them. For example, StaticContentSitemap.jsp:

The jndi name of the JDBC data source for example, <jndiName>jdbc/jndiName</jndiName>. It is only used when you run the crawler through URL. When this parameter is specified, the crawler can use that data source to update the database after the crawling finishes.
The filters configuration file determines whether URLs are included or ignored by the site content crawler.
You can update the filters configuration file by using regular expressions to include or ignore values.
Important: You must update the filters configuration file to include your WebSphere Commerce host name.
The default sample values contain ignores such as excluding URLs containing email or FTP links, or pages that require logging on to the site.
The site map, which is used by web browsers and external search engines, contains pointers to the different starter store pages
The static site map contains pointers to the static content files that are in the WebSphere Commerce database.
The URL that is passed from the configuration file to the site content crawler is:
You must update the static site map file to include your additional static content files that are in the WebSphere Commerce database.

This file is used only by the site content crawler.

Site content crawler manifest files
The site content crawler manifest.txt output files are comma-separated values (CSV) formatted documents that contain generated information.
  1. The manifest file that indicates which folder contains the downloaded site content files. It contains the following columns:
    The time stamp for the column.
    Directory path
    The counter directory path.
    Initial location URLs
    The initial URLs separated by a comma.
  2. The manifest file that contains the mappings of downloaded files to URLs. It contains the following columns:
    The ID that distinguishes each file in the document. For example, a simple sequence.
    The relative URL to the current store, or full URL pointing to external resources.
    Local file path
    The file path, either in full format or relative format, of the stored site content.
    The content type of the file for example, text/html.
    The encoding of the file, if it is a text-based file.