F. Configuring connectivity between Unica Campaign listener server and Hadoop cluster

About this task

The loader Script is needed to copy temporary data file at <Campaign_Home>/partitions/partition[n]/tmp to Impala Server which is created during flowchart execution.

Procedure

  1. We are making use of pscp (Putty utility) to connect to Cloudera Hadoop Cluster from Unica Campaign server and upload the data files on Hadoop.Note: Here the PSCP utility needs password in argument in plain text.
  2. Configuration for LoaderPreLoadDataFileCopyCmd: LoaderPreLoadDataFileCopyCmd = C:\Unica11\Campaign\bin\CopytoHadoop.bat <DATAFILE> /tmp/
  3. Configuration for LoaderPostLoadDataFileRemoveCmd: LoaderPreLoadDataFileCopyCmd = C:\Unica10\Campaign\bin\removeFromHadoop.bat <DATAFILE>
  4. Configuration for LoaderPostLoadDataFileRemoveCmd =LoaderPostLoadDataFileRemoveCmd= C:\Unica10\Campaign\bin\removeFromHadoop.bat <DATAFILE> /tmp/

Example

This is a sample script . You can write your own batch script:
Sample Script of copytoHadoop.bat:
=========================================
REM This script is used to copy a file from Campaign to any location on the 
Impala-based Hadoop system. The location must a valid HDFS location
@echo off
REM Operating System location
pscp -pw <password> "%1" <username>@<Hadoop Node with Impala>:"%2"
set filepath="%1"
for /F "delims=" %%i in (%filepath%) do set basename="%%~nxi"
for /F "delims=" %%i in (%filepath%) do set filename="%%~ni"
for /F "delims=" %%i in (%filepath%) do set fileExtension="%%~xi" 
REM Moving Campaign data with name containing special character is prohibited,
 so extracting
plink -pw <password> -ssh <username>@<Hadoop Node with Impala> "mv %2/%baseName% %2/%fileName%" 
plink -pw <password> -ssh <username>@<Hadoop Node with Impala> "hadoop fs -put -f %2/%fileName% %2" 
plink -pw <password> -ssh <username @<Hadoop Node with Impala> "hadoop fs -mv 
%2/%fileName% %2/%baseName%"
plink -pw <password> -ssh <username @ Hadoop Node with Impala> "hadoop fs -chmod 0777 %2/%baseName%
"=========================================
Data files are copied from Unica Campaign to a temp folder on the Imapala-based Hadoop system. You must use the SSH "rm" command to remove the temporary data file.
  1. Remove temp data from Hadoop file System.
  2. Remove temp data from OS file System.
Note:
  1. For LoaderPostLoadDataFileCopyCmd and LoaderPostLoadDataFileRemoveCmd to work, the Hadoop System should have TGT and have access to write, modify and remove files from Hadoop File System.
  2. The above configuration is Sample configuration, you can refer Unica Campaign Installation Guide for details.