Adam Jones - Systems Engineer
Recently we migrated a relatively small company from a POP3 email service to their own Microsoft Exchange 2010 server. Over the course of two days we imported approximately 25 mailboxes which grew the message store database to around 16GB. On the fifth day we noticed that the disk drive on our brand new mail server was completely full. We discovered that the Exchange transaction logs had grown to an astonishing 189GB in just three or four days!
The first step was to immediately remedy the storage issue by dismounting the mailbox database and using the eseutil /mh command to verify that the database had shutdown cleanly and that there were no more log files to be played into the database. This turned out to be true which didn't really help us come to a conclusion. We decided to move the massive amount of logs to an alternative storage location just in case.
With the database mounted and the storage situation taken care of (for now) it was time to begin monitoring the logs. I began checking on the server every few hours over the weekend and to my delight it appeared that the logs had settled down and all was well. As users began to trickle in Monday morning I quickly found out that I was very wrong. I watched transaction logs be created at the rate of one log file every 1 to 3 seconds! After doing a little research I was pointed in the direction of ExMon. This is the Exchange User Monitoring utility that was written by MS guys a few Exchange versions back but it continues to be supported to this day. Armed with the user monitoring tool, I was able to watch performance activity as the various mailboxes were accessed and manipulated.
A few patterns became evident and two very important columns in the tool helped me resolve our problem. One column being CPU% and the most important being Log Bytes. CPU% is the store CPU percentage consumed by the user. This can reach very high numbers upon opening Outlook and during a send/receive action but it should not constantly be in the 90%-100% range for a single user. The other tipoff was the Log Bytes per user. As a user receives a message or takes action on a message by moving it to a folder or deleting it etc., Exchange will create a transaction of this event and store it in a log file. If a user has an excessively large number of log bytes written along with an excessively high CPU% over several refresh periods, you can be sure that they are your trouble mailboxes.
It turns out that four of our users had some imported messages that were corrupted and therefore stuck on synchronizing to the database. Each time the message begins to sync a transaction is recorded. Since the message failed to synchronize it attempts again and again which creates a constant loop. We determined that all of the necessary mail had already completed the import process so the resolution was simple. We rebuilt each problem users Outlook profile which downloaded all of the "good" mail that was successfully synchronized after the initial PST import.
Tips for using the ExMon tool can be found here.