X owa error microsoft exchange data storage storagetransientexception

I’ve inherited a hybrid Exchange 2010/2013 environment where 2010 has not been fully decommissioned due to public folders. All users have been migrated to 2013 (CU12). Slowly, a couple users every couple weeks will start being unable to open their mailboxes in Outlook, webmail, or ActiveSync coinciding with StorageTransientException errors (event ID’s 14035, 4009, 11008). Sometimes, I’m able to fix their mailboxes by doing repair requests, followed by mailbox moves or offline moves. When all of those attempts fail I’ve been having to recreate their mailboxes and restore all items from backup, which is painful because these are all larger mailboxes. We are trying to get all of our users on a newer retention policy that will send their older mail to online archives, but have paused doing that as that is when we first noticed issues.

In fact, the only common thread I can find is that all of these users have been around since Exchange 2010 or possibly 2003. It has not affected any mailboxes that were created on 2013 yet. It has affected users on nearly all of my databases, which are hosted on an EMC SAN with individual LUNs for each DB.

Perfmon and ExBPA indicate no issues related to storage. The two mailbox servers at the primary site are not overtaxed and each have 8 cores and 32 GB of ram. I have been hoping that decommissioning 2010 will free up some resources but that has been delayed on our end.

I have two separate Microsoft tickets open where they agree there is an issue but claim their team is only break/fix and will not perform root cause analysis.

Does anyone have any ideas of how I can track down the cause of this? I’ve tried moves, throttling policies, clearing activesync devices, nearly everything. Here are some errors seen from the mailbox move logs and the event logs:

Transient error StorageTransientException has occurred. The system will retry (5/60).

Exception returned is: Microsoft.Exchange.Data.Storage.StorageTransientException: Cannot query rows in a table. —> Microsoft.Mapi.MapiExceptionTimeout: MapiExceptionTimeout: Unable to query table rows. (hr=0x80040401, ec=-2147220479) Diagnostic context: Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=184] Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x80040401][length=194][latency=60109]

user@contoso.com;3;[{LRT=};{LED=432 4.2.0 STOREDRV.Deliver.Exception:StorageTransientException.MapiExceptionTimeout; Failed to process message due to a transient exception with message Cannot open mailbox

When you try to open the Exchange Admin Center (EAC) to manage your Exchange Server or servers, you may get an error, such as Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException. The complete error message reads as:

X-ClientId: T1T8 – TACB – 1UYP – EFDPEZGA
X-OWA-Error: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
X-OWA-Version: 15.1.225.42
X-FEServer: BRA-EXCHANGE
X-BEServer: BRA-EXCHANGE
Date: 18/02/2017 22:07:53

From the error message, it seems that the error has occurred due to an issue while connecting to the database or mailbox permissions.

In this post, we’ll be discussing the ways to identify the issue and fix this error.

How to Fix the Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException Error?

Here are the steps to fix the Error: Microsoft Exchange Data Storage ConnectionFailedTransientException

Step 1: The first thing you need to do is ensure that the emails are working. If the users are able to send and receive emails using their locally installed Outlook, this will ensure that the business is still running. However, if users cannot send or receive emails, then you might have some pressure in getting this solved at the earliest possible time.

Step 2: Now, go back and track the changes from when the web apps were working till you identified that the web apps started showing the error. It’s good to keep a log of what was changed so that when an issue occurs, you can identify the changes, configurations, or installation of third-party application which would have caused the issue. In this log, Windows patches installations are also listed.

Step 3: Once you check the Windows Logs (see above), you can then go and check the related errors in the Exchange section under Microsoft, which is in the Applications and Services section. Although any issue with services and others will be shown in the Windows Logs (see the below section), you will find Exchange Server specific event messages that might help with the investigation.

Step 4: The next thing to do is ensure that the services of Exchange Server are running and have started without any issues. After this, you should check the event viewer for any related error message or any information which would help in identifying the issue. Sometimes, Exchange Server shows an error which is too generic. But when viewing the entries in the Event Viewer, you can find the cause of the issue or at least a direct indication to the culprit. Event Viewer is your first choice to look before starting to troubleshoot.

Step 5: You need to establish where the issue is by trying to understand if this error is specific to a particular mailbox, all mailboxes, or to the Exchange Admin Center (EAC). Try to get a sample of users and get them to log in their Outlook Web Access (OWA). If you have more than one Exchange Admin user (which is strongly suggested), try to log into the Exchange Admin Center (EAC) to ensure that the problem is global or with a specific user.

You can also check the status of the mailbox database where the affected mailboxes are situated. To check the database, first you need to check if there are any issues with the copy status. You can check this by running the below command:

Get-MailboxDatabaseCopyStatus

This will give an indication if there is a problem with the copy status of the database and if the database is mounted or not. If the database is not mounted, you can try to mount the database via the Exchange Management Shell (EMS) or by using the Exchange Admin Center (EAC), if another user is working. To do it via PowerShell, you can use the following command:

Mount-Database MyDatabase

Step 6: Now, if all goes well, the database mounts and the users will again be able to connect with the specific database. But if the database fails to mount, there could be signs of corrupted database or logs. You can try to repair the database by using the native utility – EseUtil and perform a soft recovery, which will basically run a quick recovery of the database. This is ideal to start with as it might fix minor corruption issues which are stopping the database to mount. After this is successful, try to mount the database. If this is unsuccessful, you might try the hard recovery but be cautious as this would result in data loss. The other option is to restore the database from the backup. But this would mean that any data from the backup till the error showed up will be lost.

Final Thoughts

The best alternative to fix the Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException error and retain the data, with the least downtime possible, is to use Stellar Repair for Exchange. This application can open any Exchange Server version database. You don’t need Exchange Server to open the database as the application can be installed on a Windows 10 machine. You can export mailboxes granularly from the database to PST and other file formats. You can also export directly to a live Exchange Server database. Stellar Repair for Exchange helps you recover the Exchange Server database data in no time and with no data loss.

Some users are receiving this error randomlly at OWA webmail:

Request 
Url: https://xxx:443/owa/default.aspx 
User host address: xxx 

Exception 
Exception type: Microsoft.Exchange.Data.Storage.StorageTransientException 
Exception message: There was a problem accessing Active Directory. 

Call stack 
Microsoft.Exchange.Data.Storage.ExchangePrincipal.FromUserSid(ADRecipientSession recipientSession, SecurityIdentifier userSid) 
Microsoft.Exchange.Clients.Owa.Core.OwaWindowsIdentity.CreateExchangePrincipal() 
Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.GetExchangePrincipal(OwaContext owaContext, ExchangePrincipal& exchangePrincipal) 
Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.PrepareRequestWithoutSession(OwaContext owaContext, UserContextCookie userContextCookie) 
Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.InternalDispatchRequest(OwaContext owaContext) 
Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchRequest(OwaContext owaContext) 
System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 

Inner Exception 
Exception type: Microsoft.Exchange.Data.Directory.ADTransientException 
Exception message: Could not find any available Global Catalog in forest xxx.local. 

Call stack 
Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) 
Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType) 
Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, ADObjectId& rootId) 
Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId) 
Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator) 
Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor) 
Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties) 
Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindBySid(SecurityIdentifier sId) 
Microsoft.Exchange.Data.Storage.ExchangePrincipal.FromUserSid(ADRecipientSession recipientSession, SecurityIdentifier userSid)

Also I have notice that users with Outlook and Thunderbird with Exquilla, sometimes are unable to send mails, but doesnt show any conclusive error, just an error message saying that it is unable to send.

At least in server C:WindowsSystem32LogFilesHTTPERR with IIS logs, I dont have anything related.

Also at event logs I have several logs about this issue:

Event ID 2080: MSExchange ADAccess

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=3028). Exchange Active Directory Provider has discovered the following servers with the following characteristics: 
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version) 
In-site:
xxx.xxx.local   CDG 1 6 7 1 0 1 1 7 1
 Out-of-site:

Event ID 4001: MSExchange System Attendant Mailbox

A transient failure has occurred. The problem may resolve itself in awhile. The service will retry in 56 seconds. Diagnostic information:

Could not find any available Global Catalog in forest xxx.local.
Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Global Catalog in forest xxx.local.
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)
   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, ADObjectId& rootId)
   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
   at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindByLegacyExchangeDN(String legacyExchangeDN)
   at Microsoft.Exchange.Servicelets.SystemAttendantMailbox.Servicelet.Work()

Event ID 1022: MSExchangeIS Mailbox Store

Logon Failure on database "First Storage GroupMailbox Database" - Windows account NT AUTHORITYNETWORK SERVICE; mailbox /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=user1.Error: -2147221231 Client Machine: xxx Client Process: edgetransport.exe Client ProcessId: 0 Client ApplicationId: Client=HUB 
Text
Event ID 1: MSExchange Autodiscover
Unhandled Exception "Could not find any available Global Catalog in forest xxx.local."Stack Trace:    at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, ADObjectId& rootId)   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId)   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator)   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)   at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindBySid(SecurityIdentifier sId)   at Microsoft.Exchange.Autodiscover.Providers.Outlook.OutlookAutoDiscoverProvider..ctor(RequestData requestData)

Event ID 1: MSExchange Autodiscover

Unhandled Exception "Could not find any available Global Catalog in forest xxx.local."Stack Trace:    at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, ADObjectId& rootId)   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId)   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator)   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)   at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindBySid(SecurityIdentifier sId)   at Microsoft.Exchange.Autodiscover.Providers.Outlook.OutlookAutoDiscoverProvider..ctor(RequestData requestData)

Event Log order:

Anyone?