onstat -g rqm: Prints statistics for RQM queues

Prints statistics and contents of the low-level queues (send queue, receive queue, ack send queue, sync send queue, and control send queue) managed by the Reliable Queue Manager (RQM).

The RQM manages the insertion and removal of items to and from the various queues. The RQM also manages spooling of the in-memory portions of the queue to and from disk. The onstat -g rqm command displays the contents of the queue, size of the transactions in the queue, how much of the queue is in memory and on disk, the location of various handles to the queue, and the contents of the various progress tables. You can choose to print information for all queues or for just one queue by using one of the modifiers described below.

If a queue is empty, no information is printed for that queue.

The onstat -g rqm can take an optional modifier. The following table describes the values for the modifier.
Table 1. Values for the modifier to the onstat -g rqm command
Modifier Action
ACKQ Prints the ack send queue
CNTRLQ Prints the control send queue
RECVQ Prints the receive queue
SBSPACES Prints detailed statistical information about the sbspaces configured for CDR_QDATA_SBSPACE.
SENDQ Prints the send queue
SYNCQ Prints the sync send queue
FULL Prints full information about every in-memory transaction for every queue
BRIEF Prints a brief summary of the number of transactions in each of the queues and the replication servers for which the data is queued Use this modifier to quickly identify sites where a problem exists. If large amounts of data are queued for a single server, then that server is probably down or off the network.
VERBOSE Prints all the buffer headers in memory

When you specify a modifier to select a specific queue, the command prints all the statistics for that queue and information about the first and last in-memory transactions for that queue. When you select the SBSPACES modifier, the command prints information about the sbspaces being used for replication, including how full those sbspaces are.

The other modifiers of the onstat -g rqm command are used primarily as a debugging tool and by Technical Support.

The output for the SENDQ modifier contains the following sections:
  • The current statistics section (Transaction spool name through Pending Txn Data): Contains information about the current contents of the queue, such as how many bytes are contained in the queue, how many transactions are in the queue, how many transactions are currently in memory, how many have been spooled to disk, how many exist only on disk, and so on. The Insert Stamp field value is used to maintain the order of the transactions within the queue. The Size of Data in queue field shows the size of the queue when combining the in-memory transactions with the spool-only transactions. The Pending Txn Buffers field contains information about transactions that are in the process of being queued into the send queue.
  • The historical statistics section (Max Real memory data used through Total Txn Lookups): contains a summary of what has been placed in the queue in the past. The Max Real memory data used field contains the largest in memory size of the queue. The Total Txn Recovered field shows the transactions that existed only in the spool when the server was started. The Total Txns deleted field shows the number of transactions that have been removed from the queue. The Total Txns duplicated field contains the number of times attempted to queue a transaction that had already been processed. The Total Txn Lookups field is a counter of the number of times that an Enterprise Replication thread attempted to read a transaction.
  • The Progress Table section: contains information on what is currently queued, to which server it is queued for, and what has been acknowledged from each of the participants of the replicate. The first part of the progress table section is a summary. Below the summary section is a list of the servers and group entries that contain what is currently queued for each server, what has been sent to the remote server, and what has been acknowledged from the remote server. The contents of the ACKed and Sent columns contains the key of the last transaction that was acknowledged from the remote server or sent to that server. The key is a multi-part number consisting of source_node/unique_log_id/logpos/incremental number. The transaction section contains the first and last transaction in the queue that are currently in memory. The NeedAck field shows from which server the transaction is waiting for an acknowledgment. You can use this bitmap mask with the output from the onstat -g cat command to determine the name of the server which server Enterprise Replication is waiting on for an acknowledgment.
  • The Transverse handle section: contains the position within the queue that any thread is currently processing. Each thread that attempts to read a transaction from the queue, or to place a transaction into the queue must first allocate a handle. This handle is used to maintain the positioning within the queue.

The following example shows output for the onstat -g rqm SENDQ command.

> onstat  -g rqm SENDQ


CDR Reliable Queue Manager (RQM) Statistics:

RQM Statistics for Queue (0xb956020) trg_send
 Transaction Spool Name: trg_send_stxn
 Insert Stamp: 9/0
 Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK
 Txns in queue:             0
 Log Events in queue:       0
 Txns in memory:            0
 Txns in spool only:        0
 Txns spooled:              0
 Unspooled bytes:           0
 Size of Data in queue:     0 Bytes
 Real memory in use:        0 Bytes
 Pending Txn Buffers:       0
 Pending Txn Data:          0 Bytes
 Max Real memory data used: 385830 (4194304) Bytes
 Max Real memory hdrs used  23324 (4194304) Bytes
 Total data queued:         531416 Bytes
 Total Txns queued:         9
 Total Txns spooled:        0
 Total Txns restored:       0
 Total Txns recovered:      0
 Spool Rows read:           0
 Total Txns deleted:        9
 Total Txns duplicated:     0
 Total Txn Lookups:         54

 Progress Table:
        Progress Table is Stable
                On-disk table name............:         spttrg_send
                Flush interval (time).........:         30
                Time of last flush............:         1207866706
                Flush interval (serial number):         1000
                Serial number of last flush...:         1
                Current serial number.........:         5

Server    Group Bytes Queued      Acked                     Sent
------------------------------------------------------------------------------
    20  0xa0002      12 efffffff/efffffff/efffffff/efffffff -  a/e/1510a1/0
    20  0xa0003                  0 a/e/4ca1b8/0             -  a/e/4ca1b8/0
    30  0xa0004                  0 a/e/4ca1b8/0             -  a/e/4ca1b8/0
    20  0xa0004                  0 a/e/4ca1b8/0             -  a/e/4ca1b8/0
    20  0xa0001                  0 a/d/6e81f8/0             -  a/d/6e81f8/0


First Txn (0x0D60C018) Key:  1/9/0x000d4bb0/0x00000000
 Txn Stamp: 1/0, Reference Count: 0.
 Txn Flags: Notify
 Txn Commit Time: (1094670993) 2004/09/08 14:16:33
 Txn Size in Queue: 5908
 First Buf's (0x0D31C9E8) Queue Flags: Resident
 First Buf's Buffer Flags: TRG, Stream
 NeedAck: Waiting for Acks from <[0004]>
 No open handles on txn.

 Last Txn (0x0D93A098) Key:  1/9/0x00138ad8/0x00000000
 Txn Stamp: 35/0, Reference Count: 0.
 Txn Flags: Notify
 Txn Commit Time: (1094671237) 2004/09/08 14:20:37
 Txn Size in Queue: 6298
 First Buf's (0x0D92FFA0) Queue Flags: Resident
 First Buf's Buffer Flags: TRG, Stream
 NeedAck: Waiting for Acks from <[0004]>
 Traverse handle (0xbca1a18) for thread CDRGeval0 at Head_of_Q,  Flags: None
 Traverse handle (0xb867020) for thread CDRACK_1 at Head_of_Q,  Flags: None
 Traverse handle (0xbcbd020) for thread CDRGeval1 at Head_of_Q,  Flags: None
 Traverse handle (0xbd08020) for thread CDRGeval3 at Head_of_Q,  Flags: None
 Traverse handle (0xbe511c8) for thread CDRGeval2 at Head_of_Q,  Flags: None
 Traverse handle (0xbe58158) for thread CDRACK_0 at Head_of_Q,  Flags: None

The following output is an example of the onstat -g rqm SBSPACES command.

onstat -g rqm sbspaces

Blocked:DDR 


RQM Space Statistics for CDR_QDATA_SBSPACE:
-------------------------------------------
name/addr      number    used      free   total    %full   pathname
0x46581c58     5         311       1      312      100     /tmp/amsterdam_sbsp_base
amsterdam_sbsp_base5     311       1      312      100     

0x46e54528     6         295       17     312      95      /tmp/amsterdam_sbsp_2
amsterdam_sbsp_26        295       17     312      95      

0x46e54cf8     7         310       2      312      99      /tmp/amsterdam_sbsp_3
amsterdam_sbsp_37        310       2      312      99      

0x47bceca8     8         312       0      312      100     /tmp/amsterdam_sbsp_4
amsterdam_sbsp_48        312       0      312      100

In this example, the sbspaces are all either full or nearly full.