Installing interim fixes using the roll out update process

Roll out update is a means to achieve high availability when applying interim fixes to your WebSphere Commerce application which is deployed to a clustered environment. Not all nodes in the cluster need to be shut down during the process. The deployment manager chooses one node and stops the server only on that specific node. After synchronizing the updates for the node, the server is started again. The process is repeated for each node in the cluster.

Before you begin

Whether or not roll out update is safe to use with a deployment highly depends on the deployment payload and its payload dependencies. During the roll out update process, the cluster will be in a state where some nodes have been updated, but others have not. Deployment payloads that result in a code mismatch between the client/browser and the WebSphere Commerce server are susceptible to failure or unexpected behavior. You should also consider any database changes that client/browser code depends on. If the client/browser and the server are expecting a change to be present in the database, such as a new command in the CMDREG table, then the database should be updated before any code uses it. Performing roll out update on a test cluster while accessing the cluster might help reveal any problems.
In order to use roll out update, the following conditions must be met:
  1. The interim fixes that you want to apply must be marked with This fix is roll out update enabled.
  2. WebSphere Commerce Update Installer version 7 or higher is installed.
  3. The WebSphere Commerce application is federated and clustered with two or mode nodes.
  4. Test the interim fixes on a non-production environment first.
  5. To avoid save conflicts, ensure that the roll out update is not run at the same time as any other actions that also require an EAR update. See Troubleshooting: Save conflict error for more information.

About this task

  1. The WebSphere Commerce application is active when the interim fixes are applied. The roll out update runs for a longer time when compared to the regular process of applying fixes. The nodes in the cluster are being updated sequentially rather than in parallel. This means that some nodes in your cluster have updated code while other nodes do not. Ensure this condition is feasible for your environment especially if you are using customized code.
  2. During the roll out update process, if a lower failover time is required OR if the following message is posted in the browser window:
    Servlet has become temporarily unavailable for service
    see the plugin-cfg.xml file and make the following changes to decrease the failover time:
    • ServerIOTimeout: Reduce to 30
    • ConnectTimeout: Reduce to 15
    Also consider modifying the following properties:
    • WebContainer custom property: com.ibm.ws.webcontainer.ServletDestroyWaitTime
    • JVM Custom Property: com.ibm.ejs.sm.server.quiesceTimeout
    • JVM Custom Property - com.ibm.ejs.sm.server.quiesceInactiveRequestTime
  3. If you see application exceptions in the server logs during the roll out update process, the default 60 second wait time might not be enough. Use the com.ibm.websphere.management.application.updatesync.appExpansionTimeout custom JVM property to increase the wait time.
  4. Do not use the roll out update process to:
    • Apply WebSphere Commerce fix packs
    • Apply WebSphere Application Server fix packs
    • Apply WebSphere Application Server interim fixes
    • Uninstall any WebSphere Commerce or WebSphere Application Server interim fixes
Note: You cannot uninstall an interim fix using roll out update. If you need to uninstall an interim fix, follow the steps in Uninstalling WebSphere Commerce fix packs.

Procedure

To enable the roll out update to apply interim fixes, complete the following steps:

  1. Download this earUpdateControl.properties file into one or both locations depending on the type of instance you are updating:
      • WebSphere Commerce instance:
        • SolarisLinuxAIXWindowsWC_installdir/instances/instance_name/properties
        • For IBM i OS operating systemWC_userdir/instances/instance_name/properties
      • WebSphere Commerce payment instance:
        • SolarisLinuxAIXWindowsWC_installdir/payments/pay_instance_name/properties
        • For IBM i OS operating systemWC_userdir/payments/pay_instance_name/properties
  2. Edit the earUpdateControl.properties file by setting all values to true instead of false.
  3. Run the WebSphere Commerce Update Installer and apply the interim fixes to the WebSphere Commerce product.
  4. Run the WebSphere Commerce Update Installer and apply the interim fixes to the WebSphere Commerce instance or payments instance. See the example section for roll out update information.
  5. After the installation process has completed and the WebSphere Commerce update installer returns a success message, the roll out update is complete. Continue to apply any other interim fixes.

    If you see an error during the update, increase the com.ibm.websphere.management.application.updatesync.appExpansionTimeout timeout value. Increase in 5 minute increments, do not use a value greater than 60 minutes.

    [8/25/09 16:16:31:709 GMT+08:00] 00000017 TreeBuilder W ODCF0002E: Exception: java.lang.reflect.InvocationTargetException
.
Caused by: org.eclipse.jst.j2ee.commonarchivecore.internal.exception.NestedJarException: 
IWAE0008E An error occurred reading InitializationServlet.war from /usr/WebSphere/AppServer/profiles/AppSrv01/config/cells/wci156pCell01/applications/WC_demo.ear/deployments/WC_demo
.
Caused by: java.io.FileNotFoundException: /usr/WebSphere/AppServer/profiles/AppSrv01/installedApps/WC_demo_cell/WC_demo.ear/InitializationServlet.war 
(A file or directory in the path name does not exist.)

Example

Indications that the roll out update is deploying the interim fix:
  • Check the WC_installdir/logs/update/actions/install/exportEar_WC_instance_name.log.
    Since the WebSphere Commerce application should be running during the process of fix installation, if the log contains messages similar to the following messages:
    [wsadmin] WC_demo on node wcs204Node01 with process member01 stopped
  [wsadmin] WC_demo on node WC_<instance_name>_node with process server1 stopped
    ensure that the steps to set up roll out update are completed.
  • Check the WC_installdir/logs/update/actions/install/deployEar_WC_instance_name.log. The log file should contain messages related to the deployEarWithSecurity or deployEarWithoutSecurity ANT tasks. The messages are similar to the following messages:
    [wsadmin] *** Automatic node synchronization disabled
[wsadmin] *** Update of WC_instance_name from /tmp/wcupdate/WC_instance_name.ear complete
[wsadmin] *** Configuration saved
[wsadmin] *** WC_instance_name is clustered, rollout update used
[wsadmin] *** Rollout update invoked 
[wsadmin] *** Automatic node synchronization restored
  • Monitor the roll out update process by reviewing the node agent SystemOut.log files. If messages, similar to the ones listed, are found in the node agent's SystemOut.log, then roll out update has completed on that particular node. Eventually, all nodes should show indications of roll out update completion.
    0000001f NodeSync A ADMS0019I: Automatic synchronization mode is disabled.
0000001a RmmPtpGroup W DCSV1111W: DCS Stack DefaultCoreGroup at Member /cell_name\
node_name\nodeagent: Suspected another member because the outgoing connection from the
other member was closed. Suspected members is cell_name\node_name\server_name. DCS logical channel is View|Ptp.

    ......

    ADMS0003I: The configuration synchronization completed successfully.
Commerce updating file permission on application WC_instance_name
Script file is: /tmp/WC_instance_name2008.11.10.14.14.00.000321.sh
ADMN1001I: An attempt is made to launch server_name on node node_name.
DCSV1032I: DCS Stack DefaultCoreGroup at Member cell_name\node_name\nodeagent: Connected a defined member cell_name\node_name\server_name.
Commerce has finished updating file permission on application WC_instance_name
DCSV2004I: DCS Stack DefaultCoreGroup at Member cell_name\node_name\nodeagent: The synchronization procedure completed successfully. The View Identifier is (130:0.cell_name\dmgr_hostname\dmgr). The internal details are [0 0 0 0].
HMGR0228I: The Coordinator is not an Active Coordinator for core group DefaultCoreGroup.
HMGR0218I: A new core group view has been installed. The core group is DefaultCoreGroup. The view identifier is (131:0.cell_name\dmgr_hostname\dmgr). The number of members in the new view is 5.
DCSV8050I: DCS Stack DefaultCoreGroup at Member cell_name\node_name\nodeagent: New view installed, identifier (131:0.cell_name\dmgr_hostname\dmgr), view size is 5 (AV=5, CD=5, CN=5, DF=7)
DCSV1033I: DCS Stack DefaultCoreGroup at Member cell_name\node_name\nodeagent: Confirmed all new view members in view identifier (131:0.cell_name\dmgr_hostname\dmgr). View channel type is View|Ptp.
ADML0000I: The server initialization is complete. The process ID is: 51034
ADMD0023I: The system discovered process (name: <server_name>, type: ManagedProcess, pid: 51034)
ADMS0018I: The automatic synchronization mode is enabled.

What to do next

When you have completed applying all the identified fixes, remove the earUpdateControl.properties file. Removing this file avoids the problem of using the roll out update for future interim fixes that might not be enabled for the roll out update process.