Some BlackBerry smartphone user accounts fail to start and "MAPIMailbox - OpenMsgStore (0x8004011d) failed" appears in BlackBerry Messaging Agent and Application logs due to Microsoft Exchange 2010 Throttling Policy

Article ID: KB20608

Type: Support Content

Last Modified: 05-01-2014

 

Product(s) Affected:

  • BlackBerry Enterprise Server for Microsoft Exchange
  • BlackBerry Enterprise Server Express for Microsoft Exchange
CollapseEnvironment
  • BlackBerry Enterprise Server 5.0 SP1 to 5.0 SP4 for Microsoft Exchange
  • BlackBerry Enterprise Server Express for Microsoft Exchange
  • Microsoft Exchange Server 2010 RU1 to SP2 RU2
CollapseOverview

By default, client throttling is enabled in Microsoft Exchange Server 2010. Client throttling must be disabled for the BlackBerry Enterprise Server to function correctly. The following information will appear in the BlackBerry Messaging Agent (MAGT) logs:

[20400] (01/13 15:27:09.003):{0x21AC} {email@address.com} MAPIMailbox::MAPIMailbox - OpenMsgStore (0x8004011d) failed, MailboxDN=/o=CGP/ou=First Administrative Group/cn=Recipients/cn=SERVER, ServerDN=/o=AD/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=SERVER/cn=Microsoft Private MDB
[40720] (01/13 15:27:09.003):{0x21AC} MAPI call failed. Error 'The information store could not be opened.', LowLevelError 0, Component 'MAPI 1.0', Context 649

or

MailboxManager::SubsystemInitialize - OpenMsgStore (0x8004011d)

or

A review of the event application logs on the Client Access Server (CAS) used by the BlackBerry Enterprise Server should display the following event. If using a CAS array check the event logs on all CAS servers:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          2/6/2011 6:01:07 PM
Event ID:      2915
Task Category: General
Level:         Error
Computer:   CAS01.NADomain.com

Description:Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=1234). User '...BESAdmin~RCA~false' has gone over budget '...' times for component 'RCA' within a one minute period. Info: 'Policy:[Fallback], Parts:MaxConcurrency:...;'. Threshold value: '100'.

CollapseCause

Client throttling limits the amount of concurrent connections from a Messaging Application Programming Interface (MAPI) client to the Microsoft Exchange Client Access Server.

It is possible with the correctly created and assigned Client Access Server (CAS) Throttling Policy the CAS server may still be rejecting the connections.  If there is a temporary communication issue between the CAS server and its active directory domain controller the look-up of the assigned Throttling Policy to the BlackBerry Enterprise Server service account can fail. All further connections by that client will fallback to a hard coded "Fallback" throttling policy which has a RCAMaxConcurrency of 100 and this can cause agents and users to be unable to start.

CollapseResolution
To prevent this issue from happening in the future, upgrade all Microsoft Exchange 2010 servers to Microsoft Exchange 2010 SP2 Roll Up 3.  For more information, refer to Microsoft KB article 2668900.
CollapseWorkaround

Workaround 1

Point the BlackBerry Enterprise Server to a separate Client Access Server (CAS) temporarily using the hosts file until the Exchange team can determine a time when the affected Client Access Server can be rebooted.

Workaround 2

If a BESPolicy throttling policy has not already been created, then create a new throttling policy that does not limit concurrent connections to the Microsoft Exchange Server:

  1. On the Microsoft Exchange Server, click Start > Microsoft Exchange Server 2010 > Exchange Management Shell.
  2. Type New-ThrottlingPolicy BESPolicy -RCAMaxConcurrency $null -RCAPercentTimeInAD $null -RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null -EWSMaxConcurrency $null -EWSPercentTimeInAD $null -EWSPercentTimeInCAS $null -EWSPercentTimeInMailboxRPC $null -EWSMaxSubscriptions $null -EWSFastSearchTimeoutInSeconds $null -EWSFindCountLimit $null
  3. Type Set-Mailbox "BESAdmin" -ThrottlingPolicy BESPolicy.
  4. Restart the BlackBerry Controller Service.

Note: If the Microsoft Exchange Server is 2010 SP1, apply the following throttling policy settings for the CPAMaxConcurrency, CPAPercentTimeInCAS and the CPAPercentTimeInMailboxRPC in addition to the steps above:

set-ThrottlingPolicy BESPolicy -CPAMaxConcurrency $NULL -CPAPercentTimeInCAS $NULL -CPAPercentTimeInMailboxRPC $NULL

Or, if a BESPolicy throttling policy has already been created, but is still set to throttle concurrent connection, then modify the existing BESPolicy to disable throttling.

  1. On the Microsoft Exchange Server, click Start > Microsoft Exchange Server 2010 > Exchange Management Shell.
  2. Type Set-ThrottlingPolicy BESPolicy -RCAMaxConcurrency $null -RCAPercentTimeInAD $null -RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null -EWSMaxConcurrency $null -EWSPercentTimeInAD $null -EWSPercentTimeInCAS $null -EWSPercentTimeInMailboxRPC $null -EWSMaxSubscriptions $null -EWSFastSearchTimeoutInSeconds $null -EWSFindCountLimit $null
  3. Type Set-Mailbox "BESAdmin" -ThrottlingPolicy BESPolicy.
  4. Restart the BlackBerry Controller Service.

    Important: Restarting the BlackBerry Enterprise Server or its services might delay email message delivery to BlackBerry smartphones.

    Note: It might take up to 20 minutes for replication to occur and BlackBerry smartphones to start.

Workaround 3

If the preceding method does not work to reset the throttling policy, remove the existing policy and re-create a new BESPolicy.

Remove the BESPolicy with the following cmdlet:

Remove-ThrottlingPolicy -Identity BESPolicy

Note: A policy that is assigned to users cannot be removed. In order to remove a policy that is associated with any users, reassign the default policy to users and then remove the BESPolicy.

For more information on Microsoft Exchange Server 2010 throttling policies and the commands to set default policy, refer to the following Microsoft documents:

http://technet.microsoft.com/en-us/library/bb232205.aspx

http://technet.microsoft.com/en-us/library/dd351178.aspx

CollapseAdditional Information

A quick way to confirm throttling is the issue is to use IEMSTest. If throttling is the problem then IEMSTest will fail when the BlackBerry Controller service is running, but IEMSTest will succeed when the BlackBerry Controller service is stopped. This is a sure fire way to know if throttling is affecting the server.

To confirm the required policies has been disabled, follow these steps:

  1. On a computer that hosts the Microsoft Exchange Management Shell, open the Microsoft Exchange Management Shell.
  2. Type Get-ThrottlingPolicy BESPolicy
  3. Confirm the value for the following policies are null (blank):
  • CPAMaxConcurrency (SP1)
  • CPAPercentTimeInCAS (SP1)
  • CPAPercentTimeInMailboxRPC (SP1)
  • RCAMaxConcurrency 
  • RCAPercentTimeInAD 
  • RCAPercentTimeInCAS 
  • RCAPercentTimeInMailboxRPC 
  • EWSMaxConcurrency
  • EWSPercentTimeInAD
  • EWSPercentTimeInCAS
  • EWSPercentTimeInMailboxRPC
  • EWSMaxSubscriptions
  • EWSFastSearchTimeoutInSeconds 
  • EWSFindCountLimit

To check which throttling policy is applied to the BESAdmin account, follow these steps:

  1. On a computer that hosts the Microsoft Exchange Management Shell, open the Microsoft Exchange Management Shell.
  2. Type Get-Mailbox "besadmin" | fl Name,ThrottlingPolicy

Note: The custom-created throttling policy should always be used and the default throttling policy should not be modified on the Microsoft Exchange Server.

To obtain the name of the default throttling policy, follow these steps:

  1. On a computer that hosts the Microsoft Exchange Management Shell, open the Microsoft Exchange Management Shell.
  2. Type Get-ThrottlingPolicy | Where-Object {$_.IsDefault -eq “True”} | FL Identity
Note: Similar log lines can occur if mailboxes have been moved to a new Information Store and the required permissions have not been applied. Refer to KB02276 for detail on setting Exchange permissions for the BlackBerry Service account.

Disclaimer

By downloading, accessing or otherwise using the Knowledge Base documents you agree:

   (a) that the terms of use for the documents found at www.blackberry.com/legal/knowledgebase apply to your use or reference to these documents; and

   (b) not to copy, distribute, disclose or reproduce, in full or in part any of the documents without the express written consent of RIM.


Visit the BlackBerry Technical Solution Center at www.blackberry.com/btsc.