Overview of the Data Load utility

Data Load utility - user interaction diagram

The following diagram describes how the user roles interact with the data load utility:

1. The business user provides the developer with the business data.
2. The developer creates a data source template, which defines how source data must be formatted before the data is loaded.
3. The developer also creates the business object configuration file, which defines how the Data Load utility maps the input data to the business object and how to transform the business object to physical data.
4. The site administrator uses the business object configuration file to define and create the load order configuration file.
5. The site administrator sets the store and database settings in the environment configuration file.
6. The business data is formatted according to the rules of the data source template before the data is loaded to the database.
7. The formatted source data is provided to the site administrator.
8. The site administrator runs the Data Load utility along with the three configuration files (environment, load order, and business object configuration files) to load the formatted source data into the WebSphere Commerce database. After the utility runs, the site administrator also verifies the results of the load.
9. The business data is available in WebSphere Commerce to be managed by the business user.

For more information about how the Data Load process is structured, see Data Load utility architectural overview. For more information about how the Data Load utility works and what components are included in the process, see Data Load utility framework process and components.

Using the Data Load utility

To use the Data Load utility to load data into your WebSphere Commerce database, you must first configure or create the required files. To run the Data Load utility a business object configuration file, load order configuration file, environment configuration file, and data source file are all required. For more information, see Configuring and running the Data Load utility.

Best practices

Data Load utility limitations

• Newly created business objects might not show up immediately in the storefront due to caching.

  To resolve this problem, invalidate the dynamic cache manually. For more information about cache invalidation, see Removing cache entries through the Cache Monitor.

• When you update business objects, the changes might not show up in the Management Center. This issue can occur due to caching. To resolve this problem, invalidate the data cache. For more information, see Troubleshooting: Business object changes are not reflected in the Management Center.
• You can load data in only CSV formatted files. By default, you are provided only a CSV data reader to use with the Data Load utility. The Data Load utility can load only the data that is in the format that is supported by the data reader class. To load data in other formats, you must create your own data reader.

  You are provided with an XML data reader along with the existing CSV data reader. This data reader does not support all styles of XML formatted files. If your input data uses a different style of XML than the supported XML format, you must create your own data reader.

• You must configure the Data Load utility to use a business object mediator to map your input file data with the appropriate WebSphere Commerce business objects. By default, you are provided with business object mediators for loading data for business objects in the following components:
  • Catalog
  • Inventory
  • Price and Catalog filter
  • Member
  • Location
  • Commerce Composer
  • Promotions
  • Marketing

    You must apply the interim fixes for APAR JR53438.fep and APAR JR53438.fp to add the sample directory and files for loading marketing objects.

  To load other data, you can use the com.ibm.commerce.foundation.dataload.businessobjectmediator.TableObjectMediator, or you can create your own custom business object mediators.
• If your site uses WebSphere Commerce search, the delta search index might not rebuild correctly when you delete some catalog objects with the Data Load utility in delete mode. When you delete a child object of a catalog entry or category with the utility in delete mode, both the child and parent object are removed from the delta search index rebuild. This removal can cause the parent catalog entry or category to no longer be indexed or display correctly in the storefront.

  Use the utility in replace mode to delete catalog objects when your site uses WebSphere Commerce search. To delete objects with the utility in replace mode, include the value 1 for the Delete column of an object in your input file. If you do decide to delete catalog objects with the utility in delete mode, run a full index rebuild after the load operation completes.

• The TableObjectMediator does not support workspace locking.
• The member component business object mediators do not support the following actions with the Data Load utility:
  • Deleting users or organizations
  • Loading user passwords
  • Modifying the parent organization of a member
  • Modifying the distinguished name (DN) of an organization
  • Modifying the entity type of an organization

Procedures, samples, and tutorials

The following topics are available to help you learn more about the Data Load utility.

The following table lists topics by role and main task.

Role	Topic	Summary
Site Administrator	File format for Data Load input files	A data load input file contains the actual information that the Data Load utility populates into your database. Learn about how to construct such files to ensure that the loading process is successful.
	Creating data in XML format	Learn how to create a data load input file in the supported XML format.
	Creating data in CSV format	Learn how to create a data load input file in the supported CSV format
	Configuring the Data Load utility to run a file difference preprocess	If you routinely load the same generated Data Load input file from an external system or source, you can choose to run a file difference preprocess. You can run this preprocess as part of the Data Load process to ensure that you are loading only new changes when you load your newest input file.
	Configuring the CSV data reader	The CVS data reader is already provided with the Data Load utility. Learn how to configure the provided CSV data reader to change the way data is read from your CSV input files.
	Configuring the XML data reader	An XML data reader is provided by default with the Data Load utility. Learn how to configure this provided data reader to change the way data is read from your XML formatted input files.
	Configuring the component business object builder	The BaseBusinessObjectBuilder and TableObjectBuilder business object builders by defining sub elements and attribute values in the business object configuration file.
	Configuring the business object configuration file	Learn how to configure the business object configuration file that defines how to load the data into the database. In this file, you specify the implementation classes for your Data Reader, Business Object Builder, and Business Object Mediator components.
	Configuring the data load order	The data load order configuration file controls the load order of the Data Load utility. Learn how to configure your data load order file.
	Configuring a column exclusion list	You can configure a column exclusion list that causes the Data Load utility to avoid loading data into the specified columns of a table.
	Configuring the data load environment settings	Learn how to configure the environment variables that are used by the Data Load utility in your environment settings file. To configure the Data Load utility to load data into a workspace, you must add the workspace attribute to the business context in the environment settings file. Note: When you load data into a workspace, the Data Load utility respects the locking policy set in the workspace. For more information about workspace locking policies, see Workspaces locking policies.
	Data Load utility command syntax	Learn how to run the utility command that runs the Data Load process
	Verifying the results of the data load	Learn how to verify that a load operation with the Data Load utility is successfully completed.
	Loading data into workspaces using the Data Load utility	Load data into workspaces with the data load utility
	Loading values for single and multiple value attributes	Load data for attribute values for attributes with a single value and attributes with multiple values
	Reuse attribute assigned values with the Data Load utility	Configure the utility to reuse assigned values for attributes when the same value is needed for multiple catalog entries.
	Loading promotions with the Data Load utility	Load data to create or change promotions for a store.
	Loading customer segment members by email address with the Data Load utility	Load a list of email addresses to create a customer segment and populate the segment with members that are associated with the email address.
	Loading member group members by email address with the Data Load utility	Load members into a member group based on the member email addresses.
	Configuring a Data Load utility scheduler job	Use the WebSphere Commerce Administration Console to configure the Data Load utility to run as a scheduled job.
	Loading marketing objects with the Data Load utility	Load data to create marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Samples	Scenario: Initial load	The initial load is the scenario when you finish creating and configuring a new WebSphere Commerce instance; you then load your initial data into the WebSphere Commerce database.
	Scenario: Delta load	The delta load scenario is when your WebSphere Commerce server is up and running, then you want to insert, update, or delete your catalog, inventory, or price data.
	Scenario: Catalog entry update load	The utility can run in an update mode to update catalog entry data. The update mode replaces or adds data for only the columns that are specified in the input file. All other columns remain unchanged.
	Scenarios: Workspace locking for the Data Load utility	These scenarios detail the workspace locking policies that are in effect when the Data Load utility loads data into a workspace. These locking policies affect the loading of data into the workspace database.
	Sample: Setting up the Data Load utility	This sample demonstrates how to set up the Data Load utility for its first use.
	Catalog samples	These samples use CSV or XML files to demonstrate how to run the Data Load utility to load catalog data
	Inventory samples	These samples use CSV files to demonstrate how to run the Data Load utility to load inventory data.
	Member samples	These samples use CSV or XML files to demonstrate how to run the Data Load utility to load member data. These samples include a sample for loading customer segment member data.
	Price samples	These samples use CSV files to demonstrate how to run the Data Load utility to load price data.
	Location samples	These samples use CSV files to demonstrate how to run the Data Load utility to load location data.
	Store configuration samples	This sample uses a CSV or XML file to demonstrate how to load configuration name-value pair properties for a store.
	Commerce Composer samples	These samples use CSV file to demonstrate how to load Commerce Composer assets with the Data Load utility.
	Promotion samples	These samples use input files to demonstrate how to load promotions, promotion folders, and load promotions into the promotion folders.
	Sample: Loading marketing data	These samples use input files to demonstrate how to load marketing activities, campaigns, customer segments, content, attachments, and e-Marketing Spots.
Examples	Examples: Mapping catalog data	These examples use a CSV file to demonstrate how to insert, replace, or delete catalog data.
	Examples: Mapping inventory data	These examples use a CSV file to demonstrate how to insert, replace, or delete inventory data.
	Examples: Mapping pricing data	These examples use a CSV file to demonstrate how to insert, replace, or delete price data.
	Examples: Mapping catalog filter data	These examples use a CSV file to demonstrate how to insert, replace, or delete catalog filter data.
	Examples: Mapping member data	These examples use a CSV file to demonstrate how to insert, replace, or delete member data.
Developer	Data Load utility architectural overview	An understanding of how the Data Load utility works, and the components that make up the Data Load utility.
	Customizing the Data Load utility	Learn how to customize elements of the Data Load utility to create custom data readers, column handlers, business object mediators, and load extension tables. By customizing the Data Load utility you can complete the following tasks: Load data from sources that are not in a CSV or specific XML format. Resolve data from database columns that are based on input values that cannot be mapped to the column value through the default WebSphere Commerce configuration. Create a custom mediator to load data for extended or custom business objects. Load UserData into custom extension tables.
	Configuring an SFTP transport to retrieve external files for the Data Load utility	Customize an SFTP transport for a Data Load utility scheduled job to use to retrieve input files from an external source to load into WebSphere Commerce.
Tutorials	Customizing the Data Load utility for XML input	This tutorial demonstrates how to customize the Data Load utility to load data from an XML file. Note: Support for loading XML formatted files with the Data Load utility is provided by default. You do not have to complete this customization to use XML formatted input files with the Data Load utility. If your XML files use a different format than the format that is supported by default, you can use this tutorial to help you create an XML reader that supports the XML format of your files.
	Tutorial: Customizing the Data Load utility to encrypt data	This tutorial demonstrates how to customize the Data Load utility to encrypt data, such as a user password, before you load the data.