Overview of the Data Load utility

The Data Load utility is an enhanced business object based loading utility. This utility provides an efficient solution for loading information into your HCL Commerce database. You can also customize the Data Load utility to load other types of data. The Data Load utility is the recommended loading utility.

The Data Load utility supports loading data into a workspace. When you load data into a workspace you can make and preview changes to managed assets, without affecting what is running on your site.

The following general user roles interact with the Data Load utility:
Business user
Responsible for managing the business data.
Developer
Responsible for defining the data source template, business object mappings, and customizing the Data Load utility.
Site administrator
Responsible for the day-to-day operation of the Data Load utility.

Data Load utility - user interaction diagram

The following diagram describes how the user roles interact with the data load utility:

A diagram that describes how to use the Data Load utility.
  1. The business user provides the developer with the business data.
  2. The developer creates a data source template, which defines how source data must be formatted before the data is loaded.
  3. The developer also creates the business object configuration file. The business object configuration file defines how the Data Load utility maps the input data to the business object and how to transform the business object to physical data.
  4. The site administrator uses the business object configuration file to define and create the load order configuration file.
  5. The site administrator sets the store and database settings in the environment configuration file.
  6. The business data is formatted according to the rules of the data source template before the data is loaded to the database.
  7. The formatted source data is provided to the site administrator.
  8. The site administrator runs the Data Load utility along with the three configuration files (environment, load order, and business object configuration files) to load the formatted source data into the HCL Commerce database. After the utility runs, the site administrator also verifies the results of the load.
  9. The business data is available in HCL Commerce to be managed by the business user.

For more information about how the Data Load process is structured, see Data Load utility architectural overview. For more information about how the Data Load utility works and what components are included in the process, see Data Load utility framework process and components.

Running the Data Load utility

To use the Data Load utility to load data into your HCL Commerce database, you must first configure or create the required files. To run the Data Load utility a business object configuration file, load order configuration file, environment configuration file, and data source file are all required. For more information, see Configuring and running the Data Load utility.

Best practices

When you are using the Data Load utility to load data, follow these best practices:

Data Load utility limitations

  • HCL Commerce Version 9.1.12.0 or laterYou cannot use Spring Data JPA finders in Data Load components. Instead, the Data Load framework provides a class, DBPreparedStatement, with which you can issue SQL commands to fetch data directly from the database.
  • If you run the Data Load utility in parallel/multi-threaded mode, i.e. by setting multipleThreadsEnabled="true", the input CSV files must have header information. In addition, ensure that the corresponding wc-loader-object.xml file contains the property firstLineIsHeader="true". If the header information is absent, then the Data Load utility can give an error while reprocessing those CSV files that contain error records. For more information about this setting, see Data Load parallelization.
  • Newly created business objects might not show up immediately in the storefront due to caching. To resolve this problem, invalidate the dynamic cache manually. For more information about cache invalidation, see Removing cache entries through the Cache Monitor.
  • When you update business objects, the changes might not show up in the Management Center. This issue can occur due to caching. To resolve this problem, invalidate the data cache.
  • You can load data in only CSV and XML formatted files. By default, most sample configuration files that are provided with HCL Commerce are configured to load CSV files. To load XML files, you must configure the utility to use the XML data reader. Your XML input files must also use a CSV-like structure. For more information, see File format for Data Load input files.

    If you want to load data in other formats, or in an XML structure that is not supported by the default XML data reader, you must create your own custom data reader.

  • You must configure the Data Load utility to use a business object mediator to map your input file data with the appropriate HCL Commerce business objects. By default, you are provided with business object mediators for loading data for business objects in the following components:
    • Catalog
    • Inventory
    • Price and Catalog filter
    • Member
    • Location
    • Commerce Composer
    • Promotions
    • Marketing
    To load other data, you can use the com.ibm.commerce.foundation.dataload.businessobjectmediator.TableObjectMediator, or you can create your own custom business object mediators.
  • The TableObjectMediator does not support workspace locking.
  • If your site uses HCL Commerce search, the delta search index might not rebuild correctly when you delete some catalog objects with the Data Load utility in delete mode. When you delete a child object of a catalog entry or category with the utility in delete mode, both the child and parent object are removed from the delta search index rebuild. This removal can cause the parent catalog entry or category to no longer be indexed or display correctly in the storefront.

    Use the utility in replace mode to delete catalog objects when your site uses HCL Commerce search. To delete objects with the utility in replace mode, include the value 1 for the Delete column of an object in your input file. If you do decide to delete catalog objects with the utility in delete mode, run a full index rebuild after the load operation completes.

  • The member component business object mediators do not support the following actions with the Data Load utility:
    • Deleting users or organizations
    • Loading user passwords
    • Modifying the parent organization of a member
    • Modifying the distinguished name (DN) of an organization
    • Modifying the entity type of an organization
  • When you are loading userData into custom extension database tables, you cannot load data into columns that have a data type that is not supported by the utility.
  • Run a separate data load for each data type. If you perform a single data load for two different data types (for example, product, category), then it impacts the search index by producing unpredictable outcomes in it on performing certain business tasks (NRT update) using the Management Center. To resolve this, you need to then re-index the data after the successful completion of the data load into the database.

Procedures and samples

The following topics are available to help you learn more about the Data Load utility.

The following table lists topics by role and main task.

Role Topic Summary
Site Administrator File format for Data Load input files A data load input file contains the actual information that the Data Load utility populates into your database. Learn about how to construct such files to ensure that the loading process is successful.
Creating data in XML format Learn how to create a data load input file in the supported XML format.
Creating data in CSV format Learn how to create a data load input file in the supported CSV format
Configuring the Data Load utility to run a file difference preprocess If you routinely load the same generated Data Load input file from an external system or source, you can choose to run a file difference preprocess. You can run this preprocess as part of the Data Load process to ensure that you are loading only new changes when you load your newest input file.
Configuring the CSV data reader The CVS data reader is already provided with the Data Load utility. Learn how to configure the provided CSV data reader to change the way data is read from your CSV input files.
Configuring the XML data reader An XML data reader is provided by default with the Data Load utility. Learn how to configure this provided data reader to change the way data is read from your XML formatted input files.
Configuring the component business object builder The BaseBusinessObjectBuilder and TableObjectBuilder business object builders by defining sub elements and attribute values in the business object configuration file.
Configuring the business object configuration file Learn how to configure the business object configuration file that defines how to load the data into the database. In this file, you specify the implementation classes for your Data Reader, Business Object Builder, and Business Object Mediator components.
Configuring the data load order The data load order configuration file controls the load order of the Data Load utility. Learn how to configure your data load order file.
Configuring a column exclusion list You can configure a column exclusion list that causes the Data Load utility to avoid loading data into the specified columns of a table.
Configuring the data load environment settings Learn how to configure the environment variables that are used by the Data Load utility in your environment settings file.
To configure the Data Load utility to load data into a workspace, you must add the workspace attribute to the business context in the environment settings file.
Note: When you load data into a workspace, the Data Load utility respects the locking policy set in the workspace. For more information about workspace locking policies, see Workspaces locking policies.
Data Load utility command syntax Learn how to run the utility command that runs the Data Load process
Verifying the results of the data load Learn how to verify that a load operation with the Data Load utility is successfully completed.
Loading data into workspaces using the Data Load utility Load data into workspaces with the data load utility
Loading values for single and multiple value attributes Load data for attribute values for attributes with a single value and attributes with multiple values
Reuse attribute assigned values with the Data Load utility Configure the utility to reuse assigned values for attributes when the same value is needed for multiple catalog entries.
Loading promotions with the Data Load utility Load data to create or change promotions for a store.
Loading customer segment members by email address with the Data Load utility Load a list of email addresses to create a customer segment and populate the segment with members that are associated with the email address.
Loading member group members by email address with the Data Load utility Load members into a member group based on the member email addresses.
Configuring a Data Load utility scheduler job Use the HCL Commerce Administration Console to configure the Data Load utility to run as a scheduled job.
Loading marketing objects with the Data Load utility Load data to create marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Samples Scenario: Initial load The initial load is the scenario when you finish creating and configuring a new HCL Commerce instance; you then load your initial data into the HCL Commerce database.
Scenario: Delta load The delta load scenario is when your HCL Commerce server is up and running, then you want to insert, update, or delete your catalog, inventory, or price data.
Scenario: Catalog entry update load The utility can run in an update mode to update catalog entry data. The update mode replaces or adds data for only the columns that are specified in the input file. All other columns remain unchanged.
Scenarios: Workspace locking for the Data Load utility These scenarios detail the workspace locking policies that are in effect when the Data Load utility loads data into a workspace. These locking policies affect the loading of data into the workspace database.
Sample: Setting up the Data Load utility This sample demonstrates how to set up the Data Load utility for its first use.
Catalog samples These samples use CSV or XML files to demonstrate how to run the Data Load utility to load catalog data
Inventory samples These samples use CSV files to demonstrate how to run the Data Load utility to load inventory data.
Member samples These samples use CSV or XML files to demonstrate how to run the Data Load utility to load member data.

These samples include a sample for loading customer segment member data.

Price samples These samples use CSV files to demonstrate how to run the Data Load utility to load price data.
Location samples These samples use CSV files to demonstrate how to run the Data Load utility to load location data.
Store configuration samples This sample uses a CSV or XML file to demonstrate how to load configuration name-value pair properties for a store.
Commerce Composer samples These samples use CSV file to demonstrate how to load Commerce Composer assets with the Data Load utility.
Promotion samples These samples use input files to demonstrate how to load promotions, promotion folders, and load promotions into the promotion folders.
Sample: Loading marketing data These samples use input files to demonstrate how to load marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Examples Examples: Mapping catalog data These examples use a CSV file to demonstrate how to insert, replace, or delete catalog data.
Examples: Mapping inventory data These examples use a CSV file to demonstrate how to insert, replace, or delete inventory data.
Examples: Mapping pricing data These examples use a CSV file to demonstrate how to insert, replace, or delete price data.
Examples: Mapping catalog filter data These examples use a CSV file to demonstrate how to insert, replace, or delete catalog filter data.
Examples: Mapping member data These examples use a CSV file to demonstrate how to insert, replace, or delete member data.
Developer Data Load utility architectural overview An understanding of how the Data Load utility works, and the components that make up the Data Load utility.
Customizing the Data Load utility Learn how to customize elements of the Data Load utility to create custom data readers, column handlers, business object mediators, and load extension tables. By customizing the Data Load utility you can complete the following tasks:
  • Load data from sources that are not in a CSV or specific XML format.
  • Resolve data from database columns that are based on input values that cannot be mapped to the column value through the default HCL Commerce configuration.
  • Create a custom mediator to load data for extended or custom business objects.
  • Load UserData into custom extension tables.
Configuring an SFTP transport to retrieve external files for the Data Load utility Customize an SFTP transport for a Data Load utility scheduled job to use to retrieve input files from an external source to load into HCL Commerce.