"All worker threads blocked" log line appears in BlackBerry Enterprise Server Event or MAGT logs

Article ID: KB00778

Type: Support Content

Last Modified: 07-03-2012

 

Product(s) Affected:

  • BlackBerry Enterprise Server for Microsoft Exchange
CollapseEnvironment
  • BlackBerry® Enterprise Server for Microsoft® Exchange
  • Microsoft® Exchange Server
CollapseOverview

The BlackBerry® Enterprise Server debug log displays the following:

No Response All worker threads seem to be blocked

or

[50023] (05/10 15:36:53.248):{0x193C} All worker threads of one of the pools seem to be blocked (1)

CollapseResolution

The log line indicates that the threads are in a blocked state. Blocking is a normal state for a thread that uses RPC (which these threads use). It is expected to see threads enter and then leave a blocking state. It becomes abnormal when a thread remains in the blocking state for a longer than expected amount of time. In a basic sense an RPC request from a thread has the following flow:

1. Requesting thread makes and RPC request to remote server

2. Thread enters into a blocking state and waits for a response from the remote server

3. Remote server executes the request, packages the results and returns them to the requesting thread

4. Requesting thread leaves the blocking state and processes the response from the remote server

In most cases a thread will be marked as blocking because it is waiting on step 3 to complete. It should be clear from the example above that the focus of the investigation should start with the remote server as the thread is remaining in the blocking state until it receives its response. The question then becomes why is the remote server taking an unexpected amount of time to respond?

In most cases this slow response time can be attributed to poor performance of the respective Microsoft® Exchange Server. This can occur if the Exchange Server is under high load which could be caused by an unexpected high load event or from insufficient hardware in the Exchange Server to properly service the user base. In order to determine the root cause of this issue a Performance Analysis must be completed using Microsoft® Tools such as Perfmon or Perfwhiz.

Once a performance analysis is completed the root cause of the performance degradation can be identified and then steps can be taken to resolve it.

In rare circumstances this could occur due to an internal error on the BlackBerry Enterprise Server. The BlackBerry Enterprise Server actively interacts with the Microsoft Exchange Server using the RPC protocol. Due to this active nature there is some serialization (limits access to 1 thread at a time) introduced to protect data from being altered at the same time which could lead to damage to the integrity of that data. This serialization is controlled through objects such as Critical Sections and/or Mutex's.

Under rare circumstances the thread that currently owns the serialization object could experience an error and be unable to release this object. Other threads requiring access to the Microsoft Exchange server will enter a Wait state until the serialization object is free. Since an error occurred and the thread that is currently holding the serialization object will not release it, all the other threads will eventually enter a perpetual wait state and then will be marked as blocked or non responsive. In a situation such as this most messaging functionality will be negatively impacted and only a restart of BlackBerry Enterprise Server services or a reboot of the server itself will resolve the issue. Please review the following article to understand the impact:

KB04789 Restarting the BlackBerry Enterprise Server services may cause delays in message delivery

The only way to properly identify that this particular event has occurred is through analysis of a Kernel or Process Memory dump.

Disclaimer

By downloading, accessing or otherwise using the Knowledge Base documents you agree:

   (a) that the terms of use for the documents found at www.blackberry.com/legal/knowledgebase apply to your use or reference to these documents; and

   (b) not to copy, distribute, disclose or reproduce, in full or in part any of the documents without the express written consent of RIM.


Visit the BlackBerry Technical Solution Center at www.blackberry.com/btsc.