F. Configuring connectivity between Unica Campaign listener server and Hadoop cluster
About this task
Procedure
- We are making use of pscp (Putty utility) to connect to Cloudera Hadoop Cluster from Unica Campaign server and upload the data files on Hadoop.Note: Here the PSCP utility needs password in argument in plain text.
- Configuration for LoaderPreLoadDataFileCopyCmd: LoaderPreLoadDataFileCopyCmd = C:\Unica11\Campaign\bin\CopytoHadoop.bat <DATAFILE> /tmp/
- Configuration for LoaderPostLoadDataFileRemoveCmd: LoaderPreLoadDataFileCopyCmd = C:\Unica10\Campaign\bin\removeFromHadoop.bat <DATAFILE>
- Configuration for LoaderPostLoadDataFileRemoveCmd =LoaderPostLoadDataFileRemoveCmd= C:\Unica10\Campaign\bin\removeFromHadoop.bat <DATAFILE> /tmp/
Example
Sample Script of copytoHadoop.bat:
=========================================
REM This script is used to copy a file from Campaign to any location on the
Impala-based Hadoop system. The location must a valid HDFS location
@echo off
REM Operating System location
pscp -pw <password> "%1" <username>@<Hadoop Node with Impala>:"%2"
set filepath="%1"
for /F "delims=" %%i in (%filepath%) do set basename="%%~nxi"
for /F "delims=" %%i in (%filepath%) do set filename="%%~ni"
for /F "delims=" %%i in (%filepath%) do set fileExtension="%%~xi"
REM Moving Campaign data with name containing special character is prohibited,
so extracting
plink -pw <password> -ssh <username>@<Hadoop Node with Impala> "mv %2/%baseName% %2/%fileName%"
plink -pw <password> -ssh <username>@<Hadoop Node with Impala> "hadoop fs -put -f %2/%fileName% %2"
plink -pw <password> -ssh <username @<Hadoop Node with Impala> "hadoop fs -mv
%2/%fileName% %2/%baseName%"
plink -pw <password> -ssh <username @ Hadoop Node with Impala> "hadoop fs -chmod 0777 %2/%baseName%
"=========================================
Data
files are copied from
Unica Campaign to a temp folder on the
Imapala-based Hadoop system. You must use the SSH "rm" command to remove
the temporary data file. - Remove temp data from Hadoop file System.
- Remove temp data from OS file System.
Note:
- For LoaderPostLoadDataFileCopyCmd and LoaderPostLoadDataFileRemoveCmd to work, the Hadoop System should have TGT and have access to write, modify and remove files from Hadoop File System.
- The above configuration is Sample configuration, you can refer Unica Campaign Installation Guide for details.