Rolling upgrade of an online cluster with Enterprise Replication

You can perform a rolling upgrade in a high-availability cluster by temporarily converting the primary and secondary servers to stand-alone Enterprise Replication servers. The upgrade occurs without incurring any downtime because Enterprise Replication supports replication between different versions of the server software. You can use this approach to upgrade to a new major version, or to apply fix packs or interim fixes (PIDs).

Before you begin

During this procedure, you convert the primary server and the secondary servers to standalone Enterprise Replication servers. You then upgrade the software on the secondary server, stop Enterprise Replication, and then clone the server using the ifxclone command.

The following prerequisites apply when upgrading software on a cluster:

Non-logged databases are not supported.
Raw or unlogged tables are not supported.
Typed tables are not supported unless the typed table contains a primary key.
UDTs that do not support Enterprise Replication are not supported.
The CDR_QDATA_SBSPACE configuration parameter must be set on both the primary and secondary servers.
The sqlhosts file must define a server group.
The primary and secondary servers must belong to different groups.
For versions of HCL OneDB™ software earlier than 11.50.xC7, converting a primary and secondary server pair to Enterprise Replication is not supported if a table does not have a primary key.
For versions of HCL OneDB software earlier than 11.10, the sec2er command is not supported.

If you are upgrading HCL OneDB software version 11.50.xC7 or later, the sec2er command adds a primary key to any table that does not already have one defined. For large tables, adding the primary key can take a long time, during which you will not see any server activity. In addition, the sec2er command requires exclusive access to the table while adding the primary key and user transactions will be blocked from accessing the table. You might want to manually create primary keys on any large table before running the sec2er command. If you have tables that were created with the DELIMIDENT environment variable set, and the tables do not have primary keys, then you must manually create the primary keys for those tables before running the sec2er command.

About this task

There are different steps involved in the upgrade process depending on whether you are using the Connection Manager:

For high-availability clusters that use the Connection Manager to redirect user connections:
There are two options you can choose based on your requirements. You can:
- Add a new Connection Manager instance to manage user connections while the cluster is upgraded. This involves configuring a new 3.70 Connection Manager instance that supports Enterprise Replication and has corresponding changes to the sqlhosts file or other connection mechanisms for user applications. If users already have a Connection Manager group support infrastructure to manage their user connections, they can easily add the new Connection Manager for Enterprise Replication to their existing Connection Manager group to ensure that no user connection downtime occurs during the upgrade process.
- Use your existing cluster Connection Manager or Connection Manager groups throughout the upgrade process, without making any changes to the Connection Manager configuration, applications, or application connection mechanisms. This option has a 10-second down time for user connections, but if that is acceptable, you can avoid the overhead of adding a new Connection Manager instance and the configuration changes that go with it.
For clusters not using the Connection Manager to redirect user connections:
Users must take steps to move user connections to the appropriate servers during the upgrade process.

Performing a server upgrade when the Connection Manager is not in use

In this example, the terms server1 and server2 refer to server names rather than machine names.

Some additional steps are required to upgrade HCL OneDB software:

On the primary server (server1), perform a check to see whether the servers can be split into Enterprise Replication servers by running the following command:
```
cdr check sec2er -c server1 --print server2
```
The command examines the primary and secondary servers and determines if it is possible to convert them to Enterprise Replication. The command displays warnings and errors that explain conditions that may prevent the servers from converting to Enterprise Replication. The print option prints the commands that will be run when the cdr start command runs. You should fix any warnings or errors and then run the command again before performing the next step.
Run the following command from an HCL OneDB 11.70 or later server:
```
cdr start sec2er -c server1 server2
```
The sec2er command converts the primary and secondary servers into standalone servers and configures and starts Enterprise Replication. Enterprise Replication keeps the data on the servers synchronized; however, any table created after the sec2er command is run will not be replicated.
On the former secondary server (server2), upgrade the HCL OneDB software. The steps to upgrade the server are as follows:
1. Stop replication by running the following command:
```
cdr stop
```
2. Back up the logical logs:
```
ontape -a
```
3. Stop the server that contains the older version of the HCL OneDB software:
```
onmode -kuy
```
4. Log on to the server with the newly installed HCL OneDB software.
5. Start the server and let the conversion complete successfully :
```
oninit
```
6. Run the concdr.sh script to convert the syscdr database from the old software version to the new version:
```
concdr.sh old_version new_version
```
7. Start replication on the former secondary server (server 2) after it has been upgraded:
```
cdr start
```
Because Enterprise Replication supports replication between dissimilar versions of the server software, the upgraded secondary server (server2) replicates data with the former primary server (server1), so that data updates are replicated on both servers.
Move client application connections from the former primary server (server1) to the upgraded server (server2).
On the primary server (server1) use the onmode -k command to take the database server to offline mode.
```
onmode -k
```
On the former secondary server (server2) run the following command to stop Enterprise Replication:
```
cdr stop
```
You can now clone the upgraded server to set up the other secondary servers in your cluster. Clone the newly upgraded server (server2) by running the ifxclone utility on server1. Use the -d (disposition) parameter to create a standalone, RSS, or HDR secondary server. In the following examples, assume that the TCP/IP address for server1 is 192.168.0.1 on port 123, and the address for server2 is 192.168.0.2 on port 456.
- To create a standalone server:
```
ifxclone -T -S server2 -I 192.168.0.2 -P 456 -t server1 
         -i 192.168.0.1 -p 123
```
- To create an RS secondary server, specify the disposition by using the -d RSS option:
```
ifxclone -T -S server2 -I 192.168.0.2 -P 456 -t server1 
         -i 192.168.0.1 -p 123 -d RSS
```
- To create an HDR secondary server, specify a disposition of RSS then promote the RSS server to HDR:
```
ifxclone -T -S server2 -I 192.168.0.2 -P 456 -t server1 
         -i 192.168.0.1 -p 123 -d RSS
onmode -d secondary primary_server_name
```
- To create an HDR secondary server, specify the disposition by using the -d HDR option:
```
ifxclone -T -S server2 -I 192.168.0.2 -P 456 -t server1 
         -i 192.168.0.1 -p 123 -d HDR
```

At this point, the cluster is running on the upgraded server. Clients can move applications from server2 if necessary.

Performing a server upgrade when the Connection Manager is in use

Refer to the following steps when clients are using the Connection Manager without a Connection Manager group defined in the existing setup.

For this example, assume that the following Connection Manager configuration file is defined:

NAME cm1
LOG 1
LOGFILE /tmp/cm1.log

CLUSTER	cluster1
{
   INFORMIXSERVER ids1,ids2
   SLA oltp DBSERVERS=primary
   SLA webapp DBSERVERS=HDR
   SLA report DBSERVERS=(primary,HDR)
   FOC ORDER=ENABLED PRIORITY=1
}

On the primary server (server1), perform a check to see whether the servers can be split into Enterprise Replication servers by running the following command:
```
cdr check sec2er -c server1 --print server2
```
When the above command is run, the primary and secondary servers are examined to determines whether it is possible to convert them to Enterprise Replication. The command displays warnings and errors that explain conditions that might prevent the servers from converting to Enterprise Replication. The print option prints the commands that will be run when the cdr start sec2er command runs. You should fix any warnings or errors and then run the command again before performing the next step.

Reload the Connection Manager so that it directs all client connections to the primary server. Here is the revised Connection Manager configuration file:

NAME cm1
LOG 1
LOGFILE /tmp/cm1.log

CLUSTER cluster1
{
   INFORMIXSERVER ids1,ids2
   SLA oltp DBSERVERS=primary
   SLA webapp DBSERVERS=primary,HDR
   SLA report DBSERVERS=primary,HDR
   FOC ORDER=ENABLED PRIORITY=1
}

Run the following command from an HCL OneDB 11.70 or later server:
```
cdr start sec2er -c server1 server2
```
The sec2er command converts the primary and secondary servers into standalone servers and configures and starts Enterprise Replication. Enterprise Replication keeps the data on the servers synchronized; however, any table created after the sec2er command is run will not be replicated.
On the former secondary server (server2), upgrade the HCL OneDB software.
Because Enterprise Replication supports replication between dissimilar versions of the server software, the upgraded secondary server (server2) replicates data with the former primary server (server1), so that data updates are replicated to both servers.
Move client application connections from the former primary server (server1) to the upgraded server (server2).
1. Create a new Connection Manager configuration file for Enterprise Replication. The following shows a sample Enterprise Replication Connection Manager configuration file. The SLA names are same as for cm1:
```
NAME cm2
LOG 1
LOGFILE /tmp/cm2.log
MACRO list=g_server2,g_server1

REPLSET replset_1
{
   INFORMIXSERVER g_server1,g_server2
   SLA oltp DBSERVERS=${list}
   SLA webapp DBSERVERS=${list}
   SLA report DBSERVERS=${list}
}
```
  The Enterprise Replication Connection Manager must define a replicate set that includes all replicates that are generated by the sec2er command. You can see the list of replicates by running the following command:
```
cdr list repl
```
  You create a replicate set by running the following command:
```
cdr def replset replset_name repl1 repl2 ...
```
  In the above example, repl1 and repl2 are replicates created by the sec2er command.
2. Halt the cm1 Connection Manager instance and load the cm2 instance.
  Performing the above step ensures that client connections are redirected to group_2 (because server2 belongs to group_2).
  Here is a sample sqlhosts file:
```
#dbservername   nettype    hostname   servicename   options
g_server1       group      -          -             i=10
ids1            onsoctcp   host1      port1         g=g_server1
g_server2       group      -          -             i=20
ids2            onsoctcp   host2      port2         g=g_server2

oltp            onsoctcp   host1      port3 
webapp          onsoctcp   host1      port4
report          onsoctcp   host1      port5 
```
On the primary server (server1) use the onmode -k command to take the database server to offline mode.
```
onmode -k
```
You can now clone the upgraded server to the other secondary servers in your cluster.
At this point in the upgrade process, the high-availability cluster is running on the upgraded server.
Shut down the cm2 Connection Manager instance and start the cm1 instance.
On the former secondary server (server2) run the following command to stop Enterprise Replication:
```
cdr stop
```