Site content crawling in a staging environment

In a staging environment, the site content crawler is used to crawl site content, and can be configured to invoke indexing the internal site (auto-index=true) and have the content tested in staging. Once all content is approved, the site content index update can also be replicated to the repeater as part of indexprop. Again, unmanaged content asset files must be manually copied to production. When manually publishing site content from staging to production, ensure that the manifest.txt file is also replicated. This file is later used in the event of emergency updates.

For regular updates, the indexing processing is done in the staging server and then replicated to production. In this case, the manifest file does not need to be copied because it is only needed when indexing.

For emergency updates, the indexing processing is done in the production server against the repeater. In this case, the manifest.txt file must be copied.

For applying emergency fixes to site content, the following is an outline of the high-level procedures involved:

Once the site content is tested in staging, the updated asset files are manually copied to production.
In the production system, an IT Administrator updates the correct manifest.txt file to include only the set of files that must be re-indexed.
Launch the site content index rebuild script from the production system so that the repeater can be updated with the latest site content changes.
Once re-indexing the repeater is completed, the subordinate search servers in production detect these updates and immediately initiate the replication.
Optional: If the updated site content has been cached, a manual invalidation is required.

In both cases (regular updates and emergency updates), you must manually copy the updated unmanaged content asset files to production. For deleted files, the index of those deleted files must be manually deleted on the server where the indexing processing occurs by using the di-buildindex with -webcontentDelete set to true.

Elite starter store crawling

The access control policies in the Elite starter store prevent the site content crawler from accessing the static content pages, resulting in an indexing error by default. To resolve this issue, you must create another access control policy on the staging server, crawl the site, and then propagate the search index to the repeater. That is, crawl the site in the staging environment, and then copy the search index content into production to keep the pages protected.

For example, your access control policy might resemble the following snippet:


<?xml version="1.0" encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE Policies SYSTEM "../dtd/accesscontrolpolicies.dtd">
<Policies>
   <Action Name="StaticContent" CommandName="StaticContent">
   </Action>
   <ActionGroup Name="EliteStorefrontAssetStoreAllUsersViews" OwnerID="RootOrganization">
       <ActionGroupAction Name="StaticContent"/>
   </ActionGroup>
 </Policies>

Where EliteStorefrontAssetStore represents the store identifier for your Elite starter store. See Creating an access control policy to secure the new information for more information.