HDFS Adapter transaction settings

This documentation describes how transaction settings in input and output HDFS cards affect the operation of the adapter.

The only supported transaction scope for input and output cards is Map. This means that the specified transaction action takes place only once per card, per map, prior to completing the map.

OnSuccess and OnFailure actions for HDFS input and output cards are described as follows:

OnSuccess actions for input cards

The OnSuccess actions for HDFS input cards are Delete, Keep, and KeepOnContent.

  • Delete - Delete the source file.
  • Keep - Keep the source file.
  • KeepOnContent - Keep the source file if it is not empty. If it is empty, delete the file.

OnFailure actions for input cards

The only supported OnFailure action for HDFS input cards is Commit. The Commit action completes the write operation by flushing any data still staged in the adapter but not yet written to the target file.

OnSuccess actions for output cards

The OnSuccess actions for HDFS output cards are Create, CreateOnContent, !Create, and Append.
  • Create - Create the target file if target file does not exist. If the target file with the same name already exists, overwrite the file if the -OW adapter command is included, otherwise report an error.
  • CreateOnContent - Same description as Create, except the file is not created if there is no data to be written to it.
  • !Create - Do not create the target file, even if the target file does not exist. The data provided to the adapter is not sent to HDFS, but is silently ignored. If the target file exists, it is left intact.
  • Append - Append data to the target file, if it already exists. If the target file does not exist, the file is created, and data is written to the file.

OnFailure actions for output cards

The OnFailure actions for HDFS output cards are Commit, and Rollback.
  • Commit - Preserves all data that was written to the file prior to the failure.
  • Rollback - Delete the target file if the target file was created by the adapter in the current map. Otherwise, preserve the target file including any data that was written to it in the current map.