Installation Process

Installation Package

The HCL Detect installation tarball includes:

  • The HCL Detect software platform itself, comprising all of the necessary components to run HCL Detect-supported applications.
  • The pre-configured Python virtualenv environment, comprising all Python dependencies required by HCL Detect applications to run.
  • The external open source software required by HCL Detect, e.g., Apache Tomcat, Apache Kafka, among others.

Installation Process

The installation process must be carried out using the userid that will manage the HCL Detect software.

It is recommended that this user be named drive or, if using a hybrid HCL Detect/InfoSphere Streams environment, streams.

Prior to starting the actual installation, if HTTPS-secured web access to HCL Detect will be made available, a (optional) DNS entry must be configured to provide a user-friendly URL to end users as well as a self-signed or commercial SSL certificate must be on-hand as it will be required to complete the product installation.

It might be helpful to become familiar with the infrastructure used to provide HTTPS access to Detect by reading the steps outlined in the section on web proxy configuration before attempting the installation steps.

The installation process begins by un-tarring the software tarball:

$ tar xvfz drive-2.2.0-rhel07.tar.gz

The suffixes 2.2.0 denote the version you are installing and rhel07 the specific operating system (rhel07 for RedHat 7).

Extract the software:

$ mkdir -p /opt/HCL/drive/rhel07
$ tar xzvf drive-2.2.0-rhel07.tar.gz -C /opt/HCL/drive/rhel07

Again, other locations are acceptable, but /opt/HCL/drive/rhel07 is the recommended path. Now, you are ready to perform the configuration steps:

$cd /opt/HCL/drive/rhel07/HCL/bin
$ ./installer

installer is an interactive program and will guide you through specific installation and configuration choices:

HCL Detect is a commercial product, subjected to End-User License Agreement terms. A paper-based or digital
copy of these terms must have been signed and agreed by someone authorized to do so in your organization, prior to
carrying out this configuration. A non-customer specific copy of these terms is included for your reference in this
installation package (HCL/license/eula.pdf). Do you confirm that you are authorized to proceed with the
configuration based on the terms specified in your organization's own license agreement with HCL Inc. (y/n)?

As a first configuration step, you will be asked about the directory that will be used to host the HCL Detect instance, i.e., the location in the file system that HCL Detect will use to host its services their logs as well as the data that will be kept by the HCL Detect data management services.

Next, you will be asked which network interface to use for external TCP/IP traffic. You should choose the interface that provides connectivity to other hosts in the cluster (if any) and/or external services the HCL Detect application will interface with:

Please select the network interface to use for external TCP/IP traffic
(default: 'eth0'):

[0] lo: 127.0.0.1
[1] eth0 10.0.0.123
[2] tun0: 10.8.0.45

Selecting the number corresponding to the interface you want (default: '1' for interface 'eth0'): 1

The 'eth0' network interface will be used for all external TCP/IP traffic.

Once a network interface is selected, the installer will proceed to ask which transport protocol should be used between HCL Detect's web-based frontend and its backend services:

Do you want to use HTTPS-based interactions between the HCL Detect browser-based interface and its backend
([n]o: HTTP will be used (in production, a proxy such as nginx must be used to secure the client/server
communication via HTTPS); / [y]es: HTTPS will be used with a self-signed SSL certificate (recommended only for
short-term testing) (default: HTTP) (y/n)?
Warning:

HCL Detect has access to and handles potentially sensitive information:

  • Authentication information: the use of HCL Detect requires a user account. HCL Detect has its own directory of users or can, alternatively, defer to an enterprise-wide LDAP server. In either case, user passwords are employed to ensure that a user is both authenticated and authorized to use the system. While HCL Detect never stores user passwords in the clear, certain interactions require the transfering of passwords between the front end, web-based interface to the backend. Hence, encryption (through the use of HTTPS) is STRONGLY recommended.
  • Metadata and structural information about subscriber and corporate data feeds: HCL Detect carries out analytics employing data that is often private and sensitive. Once, again, interactions between its web-based interface and its backend require manipulating such data and encryption, in the form of HTTPS interactions, is STRONGLY recommended to preserve end-to-end confidentiality.

While HCL Detect is generally hosted in an internal network, never facing non-corporate users, it does integrate with other segments of the enterprise computing environment. HCL recommends that administrators take every possible precaution to protect the integrity and confidentiality of the data consumed and produced by this platform.

There are multiple possible configurations to choose from and each one has certain advantages and risks:

  • HTTP (without a proxy such as nginx securing the client/server interactions), available network-wide (STRONGLY DISCOURAGED): this is the simplest form of installing HCL Detect. Nevertheless, it is insecure as potentially sensitive information is transmitted in the clear over the network, flowing from the user's browser to the server without any encryption, including passwords (potentially, even the ones used for authenticating via enterprise-wide repositories such as a corporate LDAP server, if such an integration is eventually enabled).
    Note:

    SSL certificates

    An SSL certificates is a data file that digitally binds a cryptographic key to an organization's identity. For instance, when installed on a web server, it activates the padlock used by the browser to indicate a _secure_ connection and, hence, the HTTPS protocol in interactions between the brower and a server.

    In general, an SSL certificate is obtained from a Certificate Authority (CA) and it attests the ownership of a public key by the named subject of the certificate, providing an assurance that an interaction is occurring between a client and a properly identified server. A CA acts as a third party, trusted both by the subject (owner) of the certificate and by the party relying upon the certificate.

    The HCL Detect web service can make use of an SSL certificate to ensure that interactions between the web-based client and server are properly authenticated (i.e., to provide an assurance to the browser-based client that it is indeed speaking to an actual server) as well as encrypted, such that no sensitive information flows over between a client and server in clear text form.

    SSL certificates can be purchased from several vendors or obtained for free from organizations such as Let's Encrypt. Commercial SSL certificates are typically verified and accepted by mainstream web browsers such as Google Chrome and Mozilla Firefox.

    SSL certificates can also be provided by any entity hosting a Public Key Infrastructure (PKI). For instance, an organization's IT department might host its own internal PKI and issue self-signed certificates. These certificates work just like commercial certificates, but they will typically produce a warning or an outright rejection from web browsers, which will not recognized the PKI's CA. In such cases, upon being directed by the organization's IT department, the certificate may be added to the browser and accepted as legitimate, thus quieting down the warnings that would, otherwise, be raised every time a web server presenting it is accessed.

    HCL Detect can be configured with either type of certificate, but HCL strongly recommends that a certificate be obtained from an officially recognized commercial or non-profit CA.

  • HTTPS, available network-wide (RECOMMENDED for testing only): this configuration is deemed safe, as it minimizes the chances of a sensitive data breach. In such a configuration, the interactions between the browser-based user interface and HCL Detect's backend is carried out via HTTPS interactions (encrypted). In this case, the HCL Detect backend service (hosted by Apache Tomcat) is the direct destination of calls performed by a user's browser to the server servicing the REST APIs.

    The disadvantages of this approach are two-fold. First, it employs a self-signed certificate generated by HCL and not a commercial certificate issued by a proper Certification Authority. Second, the URL to access HCL Detect will be in the form https://drive.company.com:<port>/drive. In other words, the port where the service runs will be part of the URL. Typically, HTTPS servers bind and run on the privileged port 443 and such a port number can be omitted from the URL. While the Apache Tomcat-based Detect backend can be configured to use port 443, this configuration is not recommended since it requires running the web server as root, which opens up the possibility of a complete takeover of the host where the web server runs should an unknown (but possible) security vulnerability be exploited.

    Alternatively, it is also possible to fiddle with the Operating System settings to allow non-root-owned applications to use a privileged port. Nevertheless, such a configuration is both non-standard and complex from a system administration standpoint.

    If this configuration is chosen, the installer will ask for a non-privileged port to be used by the web server and will automatically create a certificate authority, will issue a CA certificate, and install a self-signed SSL certificate to be used by the web server as part of the installation process.

  • HTTP, available only in the localhost interface, proxied by an HTTPS web proxy (STRONGLY RECOMMENDED): this configuration is deemed the safest, as it minimizes the chances of a sensitive data breach. As with the prior configuration, the interactions between the browser-based user interface and the web proxy in front of HCL Detect's backend are HTTPS-encrypted.

    The web proxy (both Apache httpd and nginx are supported) employs a regular local HTTP connection to the HCL Detect backend and a client will interact with the proxy via HTTPS, which will in turn relay the requests to the web server.

    While interception of end-user communication with the web server is possible, this can only occur when the host where the web server is installed is compromised and root access is available to the malicious party.

    The benefit of this approach is that the web proxy (which is specifically hardened for this task), not HCL Detect's backend, runs as root. Furthermore, the web proxy's internal design is optimized for these types of interactions, the use of commercially-issued or locally-issued SSL certificates, specific SSL ciphers, as well as many other SSL-related configurations much easier to put in place.

    In this configuration, the installation of an SSL certificate as well as the configuration of the HTTPS endpoint is done by installing and configuring the web proxy, a step described in the web proxy configuration section.

Finally, an HCL Detect installation can be optionally configured with additional integration points to external software packages including IBM SPSS Modeler Solution Publisher, IBM InfoSphere Streams, as well as Oracle WebLogic Server. Each of these integration points requires a prior licensed installation for each of these software packages. When installing an HCL Detect version enabled with one or more of these integration points, additional installation steps will take place. These steps, applicable only to the specific integration points enabled in the installation, will be carried out by the installer program to ensure that it can find the proper version for each of the external packages needed by them. For instance:

Please enter the path to an existing IBM SPSS Modeler Solution Publisher
installation to use: /opt/x86_64/ibm/spss/ModelerSolutionPublisher/17.1

Please enter the path to an existing IBM InfoSphere Streams
installation to use: /opt/rhel07/ibm/streams/4.0.1.0

ERROR: failed to validate IBM InfoSphere Streams
installation at '/opt/rhel07/ibm/streams/4.0.1.0'
Product information file '/opt/rhel07/ibm/streams/4.0.1.0/.product' does not exist
Please provide a valid path to an existing IBM InfoSphere Streams installation

Please enter the path to an existing IBM InfoSphere Streams
installation to use: /opt/rhel07/ibm/streams/4.0.1.0

Please enter the path to an existing Oracle WebLogic Server
installation to use: /opt/HCL/drive/noarch/weblogic-10.3.6.0

If everything is correctly configured, a success status message will be printed out:

HCL Detect environment was successfully installed...

Now that the configuration is complete, a test can be executed. To run any HCL Detect application, the shell where the application is going to run must be configured by invoking the environment_setter script (<drive-install-location>/HCL/bin/environment_setter):

$ ./environment_setter

Once the script is source'd, several environment variables are configured and the Python virtualenv environment is activated, indicating that we are ready to start an HCL Detect application:

[drive]|$ env | sort | grep HCL
HCL_ARCH=x86_64
HCL_HOME=/opt/HCL/drive/rhel07/HCL
HCL_JAVA_PATH=/usr/lib/jvm/java
HCL_OS=rhel07
HCL_PACKAGE_VERSION=2.2.0
HCL_PYTHON_PATH=/opt/HCL/drive/307/pyenv
HCL_READLINK=readlink
HCL_TOMCAT_PATH=/opt/HCL/drive/noarch/apache-tomcat-9.0.20

The preparation of the shell must be made for every shell and session where an HCL Detect application will run. Now, we can run a test application (<drive-install-location>/HCL/bin):

[drive]|$ cd HCL/bin
[drive]|$ ./installation_tester -c -sns
INFO: this application is using '/tmp/<user>/HCL' to output and store its
code-generated assets...
.
.
Note:

We are invoking the test application installation_tester using the -c flag to indicate that the application should be re-built (just in case, it has been executed before), thus ensuring that a fresh version, based on the current installation, is executed. Normally, HCL Detect applications are only rebuilt when needed and taking such a precaution is normally not necessary.