Setting Compass Search for the first time

You need to setup and create a Compass Search “search instance” for your user database.

About this task

Compass Search Definitions and Best Practices

A Compass Search is a “search instance” for a Compass search database. This means, if you have more than one Compass user database, regardless to which Compass database set it belongs and you want to enable search for said user database, you will need to setup and create a Compass Search “search instance” for said user database. A Compass Search “search instance” will be uniquely. The uniqueness is made off the Compass database set and user name. Thus, if your database set name is “Demo” and user database name is “SAMPL”, then the unique name for this search instance is “Demo_SAMPL”.

A search instance is created at the “home” location that you chose. It is recommended that you use the same “home” for all of your search instances. This way, all of your search data are in a common location on the file system.

To ease with maintains and upgrades, a dedicated server should be used for Compass Search. This way, if the OS has to be patched or Compass has to be upgraded, they can be done without having to bring down all of Compass at the same time.

Compass Search System Requirement

Compass Search, just like with any other search engine, is disk bound more than CPU or RAM. This is because searching and especially indexing generates a lot of disk activates. To that end, having a decent fast disk is essential for search. Furthermore, having a dedicated disk for the index, instead of the OS disk, means disk activates, on which the index lives, is now dedicated to search out of the OS or other applications’ way.

Attention: Using NFS drive for the index is problematic and should be avoided. If you must use it, make sure it is based on NFSv4.

Diskspace requirement for Compass Search depends on several factors as such there is no easy way to pre-determined how much space will be needed. The reason for this is due to several variables:
  1. The number of record types and fields enabled for search. If you are indexing a small subset of your record type and fields in those record types, then the index will be smaller.
  2. The overall data size in your record types and fields enabled for search. If your indexed records contain a lot of data, then the index will be larger.
  3. The number of unique words in your data of the record types and fields enabled for search. If your data contains a lot of unique words outside the standard English words, then the index will be larger.

In general, if all record types and fields are enabled for search, it should be assumed that the index size will be 1/4 of the database size (excluding attachments if they are stored in the database).

CPU and RAM requirement for Compass Search is light. A decent CPU and 4 GB of real RAM per user database enabled for search should be sufficient.

Compass Search can be installed on Windows or Linux OS and the machine can be a VM.

Note: It is recommended that Compass Search be installed on a separate machine than the Compass server. Doing so means the search server can be managed and upgraded independently from Compass server.

Setting Compass Search for the first time

Setting up Compass Search requires some preparation. Based on your schema and organization needs, one of the main decisions you will have to make is determine which record types and fields of those record types to enable for search. In general, you will enable search on all of your records and their fields. However, there may be cases where you do not want some record types or their fields searchable. This maybe because you have record types and fields that do not add value to search results.

The process to setup Compass Search is the same as what was outlined earlier in “A quick introduction to Compass Search”. This section will follow the same steps but provide additional details.

Procedure

  1. Enabling type-ahead

    Type-ahead is an existing feature of full-text search and is supported with Compass Search. This feature let you create search capability on reference-list fields. By doing so a user can then narrow down choice-list selection to few items out of the 100’s by typing few letters. You can enable this capability on any reference-list field; however, the referenced record type and the fields must be enabled for search as part of Compass Search configuration.

    To enable this feature, first review your schema and decided which fields would benefit from this feature. Next, review the fields in the choice-list and decided which fields should be indexed and used as lookup when a user is searching in the choice-list. As an example, lets assume you will enable type-ahead on the field “Owner” of the out-of-the-box Defect Tracking schema. The “Owner” field is a reference-list to the record type “users”. The record type “users” has several fields but only some of them would be useful for type-ahead such as the “login_name”, “fullname”, “phone”, “email” and “misc_info”. If you enable type-ahead on those fields, then a user can find an item by entering any few letters that matches in those fields regardless where those letters are.

    Once you have a list of record types and their fields selected for type-ahead based on your schema and organization need, you will need to enter those into a text file, one record type per line:
    users=login_name,fullname,phone,email,misc_info
Project=Timelines
Iteration=Name

    In the above example, we are enabling type-ahead on the record types “users”, “Project” and “Iteration”. For each of those record type, we are listing the fields that will be indexed and used as look fields for type-ahead.

  2. Initializing Compass Search
    The initializing step for Compass Search will create directories, copy required files, pre-set default values and create the entity file based on your Compass’s user database for which you are enabling Compass Search. All this happens in your selected target directory. The command syntax is:
    cqperl cpsearch.pl initSearch -username admin -password "" -dbset SearchDemo -userdb SAMPL -searchHome D:\CPSearch\CPSearch.Home -solrHome D:\CPSearch\CPSolr.Home -solrPort 8984 -typeahead typeahead_cfg.txt

    Note that, if you are not enabling type-ahead, you can leave out the “-typeahead” part.

    In the above example, we are executing the “initSearch” command. This command will check to see if there isn’t an instance of search at the destination, it will then create one. If one already exists, then it will fail to complete the command. Nothing gets done if the command fails.

    The “-username” and “-password” must be a valid Compass account and the account must be Super User.

    The “-searchHome” and “-solrHome” define the destination directory into which Compass Search files will be created and configured. They can be anywhere on a file system and it is recommended that a different drive than the OS is used when setting search for production. If the directories do not exist, they will be created.

    The “-solrPort” is the port on which Apache Solr will use. This port must be available and open for access specially if Compass server is installed on a different server.

    The “-typeahead” is used to specify the type-ahead file. You can leave this out if you are not enabling type-ahead. This file will be copied into the search instance location that you specified via “-searchHome”.

    Executing the “initSearch” command will take few minutes. If there are any errors, they will be reported, otherwise the command will complete with a success message. Once the command competed, the following will happen:
    1. The directory “D:\CPSearch\CPSearch.Home\SearchDemo_SAMPL\” (based off “-searchHome”) and “D:\CPSearch\CPSearch.Solr\SearchDemo_SAMPL\” (based off “-solrHome”) will be created.
    2. In the “-searchHome” directory, the following files will be created:
      • compass_search_SearchDemo-SAMPL.properties – This file holds default settings for Compass Search such as the file locations JVM settings and internal commands used by “cpsearch” tool. You do not need to edit this file unless if you are instructed by HCL Support.
      • Entity_SearchDemo_SAMPL.txt – This file holds a list of all record types and their fields that exist in your schema. You will need to edit this file to customize it. The next section covers what need to be done to this file.
      • TypeAhead_SearchDemo_SAMPL.txt – This file holds a list of record types and fields that you want to enable for type-ahead. It contains the content of what you provided for “-typeahead”. The content could have been reformatted and cleaned up to remove any invalid record types or fields. This file will be created only if you specified the “-typeahead” argument.
    3. In the “-solrHome” directory, the following directories will be created:
      • “index” – This directory holds the Apache Solr index and index configuration files. The index and the configuration files are managed by “cpsearch” tool and Solr. There is no need for you to modify anything in this directory unless if you are instructed by HCL Support.
      • “solr” – This directory holds the Apache Solr server and the Lucene search engine. There is no need for you to modify anything in this directory unless if you are instructed by HCL Support.
  3. The Entity File
    Once the “initSearch” command completed, the “entity” file gets created in the “-solrHome” directory with the name “Entity_<dBset>_<userDb>.txt”. The entity file contains all the record types and their fields based off your schema for which you are setting up for search. You will need to edit this file to customize it in the following way:
    1. Determine which record type and fields to enable for search. In most cases, you will search on all record types and their fields. However, if your schema has a record type or set of fields that should not be searchable, then remove them from this file.
    2. Determine which field, for each record type, will become the “display field” and place an “&” in front of that field.
  4. The Type Ahead File

    If you are enabling type-ahead, using the “-typeAhead” argument, the file “TypeAhead_<dBset>_<userDb>.txt” will be created. This file will be based on content in the file you specified on the command line. If there were any invalid record types or fields, they are removed from the target type-ahead file. In general, you will not edit this file.

  5. Customizing and tunning Compass Search server
    In some cases, you may need to make changed to default setting for Compass Search such as specify the JVM memory to name some. If so, this is done by edit the file “compass_search_<dbSet>-<userDb>.properties” file. The only parameter you may need to modify in this file are the JVM settings as follows:
    • CPSolr.MaxHeapSize – This specifies the maximum heap size for Apache Solr. The default is 4 GB.
    • CPTool.IncrIndexMaxHeapSize – This specifies the maximum heap size for the incremental indexer. The default is 4 GB.
    • CPTool.FullIndexMaxHeapSize – This specifies the maximum heap size for the full indexer. The default is 4 GB.

    All of the other settings should not be modified unless if instructed by HCL Support.

  6. Deploying Compass Search

    Once “initSearch” has completed and you have customized the entity file, it is time to deploy Compass Search and enable it. The deployment process involves reading Compass records and sending them to Apache Solr for indexing. Based on the number of record types and their fields that you enabled for search and based on the total number of records each of those record types you enabled for search have, the indexing process can take anywhere from few minutes to several hours and even over 24 hours. During this time, your database server will be utilized for constant read operations and you will see increased network traffic between the database server and Compass Search server. If your database server is in high demand, you should consider indexing at off hours.

    Once “deploySearch” start, Compass Search becomes available, however, search results will not be complete until when indexing has completed.

  7. Enabling Incremental Indexer

    Once “deploySearch” has competed, you will need to enable the incremental indexer. This is done differently on Windows and Linux.

    On Windows, a new service will be created called “HCL Compass Search - <dbSet/userDb>”. Located it and start it.

    On Linux, you will need to add the following command to your systemd, init command:
    cqperl cpsearch.pl startSearchServer -dbset <your-db-set-name> -userdb <your-user-db-name> -searchHome <your-search-home> -startas continues
    For example, 
    cqperl cpsearch.pl startSearchServer -dbset SearchDemo -userdb SAMPL -searchHome /opt/cpsearch/cphome.search -startas continues