MQ Series Issues/ Problems/ Errors

                         
                            MQ Series Issues/ Problems/ Errors

1. Problem: Sender Channel in Retrying/Binding State.

Occurrence: Unable to bring the sender channel in Running status by right click and start

Solution 1. Ensure that the member is able to physically ping to RBI server from dos prompt (Open Command prompt and execute command C:\>ping 10.21.1.45 –t).

In case it is not able to ping and gives response as request timed out, please check your leased line/VSAT connectivity.

In case you get regular ping response, carry out following sequentially on sender channel (by right click on sender channel—all task---followed by):

2. Stop the sender channel. Channel status should change to stop.

3. Resolve (with option backout). It should give a message that channel resolved successfully.

4. Reset (option message sequence number 1)

5 Ping. It should give a message that channel was pinged successfully.

6. Start the channel. The sender channel status should change to Running.

2. Problem Unexpected Error AMQ4048

Occurrence When trying to Ping the Sender channel through MQ series (refer above).

Solution Stop all the Queue managers and then start only the NDS1P queue manager and then start the sender channel. It should work. Then you may restart the other queue managers. If it does not work, you may consider rebooting the server (if possible) or may contact your MQ Series vendor

3. Problem Receiver channel not starting while sender channel is in Running status

Occurrence When clicking the Question Mark on NDS Server admin screen to start the Receiver channel

Solution 1.Ensure that the Listener of NDS1P queue manager is running. (Listener can be checked by right clicking the NDS1P QMGR and going to Services. Listener can be started or stopped by right click start/ stop on the listener)

2. Stop all the Queue managers and then start only the NDS1P queue manager.

If the above does not work, call up NDS Helpdesk.

4. Problem Current Depth of UnprocQ is Non Zero

Occurrence Observed when checking the Current Depth of queues on server.

Solution Run the Unproc.bat file from the following location (in case you are not having it already, please prepare it as stated below under Remarks and save it in the stated folder)

C:\Program Files\RBIPDONDSSetUp\StartUp

Remarks The command in the Unproc.bat batch file is PollOnMsg.exe NDS1P.HL.<LUDID>.UNPROCQ NDS1P.HL.<LUD ID>.TEMP_UNPROCQ NDS1P.<LUDID>.QMGR

(where <LUDID> is the member's respective Lud ID).

5. Problem Current Depth of TEMP_ UnprocQ is Non Zero

Occurrence Observed when checking the Current Depth of queues on server.

Solution Check the box 'Process Messages from UnprocQ' on the NDS Server Admin screen (check box is under the process tabs. Check it by clicking on the small white square check box).

6. Problem Current depth in the Input Log queue/ Output Log queue is 10000 or near about 10000.

Occurrence Observed when checking the Current Depth of queues on server.

Solution Clear messages from concerned queue as 10000 is the upper limit for said queue and once it reaches the limit, NDS message flow will be affected.

Disconnect and Stop processes on the NDS Server Administration Screen and then right click the Input Log queue/ Output Log queue--- All Task ---Clear messages.

Current depth will come down to zero.

NDS server may again be connected and the processes started.

7. Problem Current Depth of TXN Q is Non Zero

Occurrence Observed when checking the Current Depth of queues on server.

Solution 1) Disconnect and stop processes on the NDS Server Admin GUI screen. Then again connect and start processes. Refresh the TXN Q continuously. The current depth may initially increase then it should come down to zero.

2) In case the current depth in TXN Queue is huge (in hundreds) then it may take time to process messages from

TXN Q. To expedite the processing of messages from TXN Q,

run the POLLON_TXN.bat file from the following location (in case you are not having it already, please prepare it as stated below under Remarks and save it in the stated folder)

C:\Program Files\RBIPDONDSSetUp\StartUp.

More than one batch files (POLLON_TXN.bat) can be run simultaneously. Each running batch file will open a separate Dos Prompt window.

Remarks The command in the POLLON_TXN.bat batch file is PollOnMsg.exe NDS1P.HL.<LUD ID>.TXN

NDS1P.HL.<LUD ID>.UNPROCQ NDS1P.<LUD

ID>.QMGR

(where <LUD ID> is the member's respective Lud ID).

Some of the more common problems that occur in this environment are:

MQ Channel Down:

An MQ Channel Down error is easy to detect either by activating the supplied situation (alert) based on an MQ Event or by querying the Channel Status attribute. The Channel Stopped event alerts only when the return code indicates that the channel stopped in error. This event provides the easiest and lowest overhead way to alert on this situation. By using the channel status attribute, the alert is enhanced by also checking the depth of the XmitQ associated with the channel and then checking the status of the channel only when there are messages in the XmitQ. This action reduces the need to check every channel and only the channels that are truly problematical are checked. So, for Channel Down problems you can have two basic alerts. One is based on the MQ event of channel stop and the return code indicating an error. The other monitors for messages in the XmitQ and the associated channel not running.

Queue Full:

On the issue of Queue Full, again there are two ways to alert on this issue. The first one is to use the Queue Full MQ event, which is a relatively easy way to detect this issue. However, to truly monitor the status of the queue depth, you must be able to monitor the actual depth of the queue. MQ events alert you when a queue depth crosses the high-depth threshold and when it is full, but these events are not resolved until the queue depth drops below the queue low threshold. So, a queue could go full and then as the problem is being rectified (maybe by starting the process to read from that queue) the queue could be in a steady state of maybe half full, with messages going on and off the queue, without there being an event generated to tell us that the queue is no longer full. So, one should set up another situation that actually monitors the depth of the queue and then when the first message comes off the queue that alert would then be automatically closed, indicating that the queue is no longer full.

Messages in the Dead Letter Queue:

Most, if not all, Queue Managers have a designated Dead Letter Queue (DLQ). The Dead Letter Queue prevents WebSphere MQ from bringing a channel down because of an undeliverable message. However, you must have a situation in place to alert you to when any messages arrive on the DLQ. One of the many enhanced metrics in the OMEGAMON XE for Messaging product is the translation of the reason code for the message being put on the DLQ. This identifies the route cause and reduces the Mean Time To Resolution (MTTR) of the problem. On resolution of the problem, use the OMEGAMON XE for Messaging product to delete or forward the message back to the original destination or some other queue.

Messages in a queue and no open processes:

If there are messages in a queue and no processes have the queue open for input or output, it is probably at least worth a warning alert. This situation can be a problem unless the queue is expected to hold messages for some future processing window. It is simple enough to set up alerts in the OMEGAMON XE for Messaging product based on these requirements. The alert can be further refined to not trigger within certain expected timeframes, when it is expected that the queue will not be opened.

Isolating MQ problems between IBM z/OS® and distributed systems:

One of the key reasons that most companies make the decision to use the WebSphere MQ product is the ability to easily connect applications across disparate systems with a common API. However, these same companies continue to manage WebSphere MQ as separate entities in the distributed and z/OS environments. To correctly diagnose MQ issues, you must be able to manage WebSphere MQ with a common set of tools across all architectures. The OMEGAMON XE for Messaging product normalizes WebSphere MQ metrics across platforms so that one person or team can easily manage WebSphere MQ from a single vantage point. Managing channels, queues, or other objects can be handled in the same way, wherever they exist, which results in a more effective level-one analysis. It no longer requires two or more distinct teams to perform level-one diagnostics and management of the WebSphere MQ environment. A platform specialist is required only for underlying platform-specific issues.

Changes in the MQ configuration:

One of the key reasons that most companies make the decision to use the WebSphere MQ product is the ability to easily connect applications across disparate systems with a common API. However, these same companies continue to manage WebSphere MQ as separate entities in the distributed and z/OS environments. To correctly diagnose MQ issues, you must be able to manage WebSphere MQ with a common set of tools across all architectures. The OMEGAMON XE for Messaging product normalizes WebSphere MQ metrics across platforms so that one person or team can easily manage WebSphere MQ from a single vantage point. Managing channels, queues, or other objects can be handled in the same way, wherever they exist, which results in a more effective level-one analysis. It no longer requires two or more distinct teams to perform level-one diagnostics and management of the WebSphere MQ environment. A platform specialist is required only for underlying platform-specific issues.

Restoring last valid MQ configuration:

OMEGAMON XE for Messaging can easily discover all WebSphere MQ objects across the enterprise and store those objects in a central repository. By having all WebSphere MQ objects in the repository, a number of unique opportunities are now available to manage these objects. The objects in the repository can be easily compared against the current physical environment either on a scheduled or on demand basis to determine if there have been any changes made to the WebSphere MQ environment. Any changes detected can then be either backed-out from the physical environment or accepted and incorporated into the central repository, depending the company's requirements.

Determining if slow performance is due to network, MQ or Message Broker:

Also, by having a record of the WebSphere MQ environment in a database, recovery of the environment for contingency planning becomes somewhat trivial. Because the WebSphere Message Broker product is based on WebSphere MQ, and is used more often for routing and transformation of MQ messages and more, using one tool to manage both products again tends to improve the MTTR of any problems that run across both products. The OMEGAMON XE for Messaging product makes it simple to detect if a slowdown in your messaging backbone is caused by the WebSphere MQ or Message Broker product.

Problems connecting to broker's queue manager:

Alerts can be set to detect such events as the Queue Manager not running, Message Broker not connected to its Queue Manager, Broker not running, or Message Flows not running or running slowly. The OMEGAMON XE for Messaging product provides a GUI environment reflecting the operation infrastructure of the Message Brokers.