Exchange app - splunk server "oversubscribed" - aggregation and parsing queues pegged

[jeff]

After installing the Splunk App for Microsoft Exchange, my "central Splunk servers" (indexers/search heads) the Aggregation Queues and Parsing Queues pegged near full for several days. After some digging, it seems to be related to the timestamp parsing for some of the powershell scripts... I was seeing a lot of errors like:

05-22-2012 10:04:17.012 -0400 WARN  DateParserVerbose - A possible timestamp match (Fri Jun 24 07:55:59 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source::Powershell|host::EXMBX2|MSExchange:2010:Folder-Usage|remoteport::6919
            19 similar messages suppressed.  First occurred at: Tue May 22 09:59:16 2012
05-22-2012 10:04:17.013 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Feb  1 01:02:12 2012). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406
            16384 similar messages suppressed.  First occurred at: Tue May 22 09:59:16 2012
05-22-2012 10:04:17.280 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Sep 20 18:23:00 2008). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406
05-22-2012 10:04:17.288 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Sep 20 18:23:00 2008). Context: source::Powershell|host::EXMBX1|MSExchange:2010:Folder-Usage|remoteport::2406

When I looked at the source for these events, there were no time stamps at all... I also did searches for the events in Splunk and came up with very few in the "current" time period... so I created a /local/props.conf file with the following attributes, the errors stopped, and the queues went back down to near zero.

[MSExchange:2010:Folder-Usage]
DATETIME_CONFIG = CURRENT

[MSExchange:2010:Mailbox-Usage]
DATETIME_CONFIG = CURRENT

[ahall_splunk · Accepted solution]

Hmmm - not sure how I missed those ones when I was doing this. There is actually a big reason why we started including a date-time stamp on the events. I'll update the scripts over the weekend and post an update next week.

[ahall_splunk]

I've just released Splunk App for Microsoft Exchange v1.1.5 that corrects this problem. The Technology Add-ons have changed to emit the timestamp.

[ahall_splunk]

You've got a mismatch between the app and the Technology Add-ons that are producing the data. The app likely got updated from v1.0 to v1.1 without the deployment of the new Technology Add-ons.

All the TA's should be the same version as the Splunk_for_Exchange app.

Specifically, the new TA's include a date/time stamp, whereas the originals used DATETIME_CONFIG=CURRENT. There was a corner case for Folder-Usage and Mailbox-Usage when the folder name or mailbox name contained something that looked like a timestamp. In that case, the event was recorded erroneously.

[jeff]

sample code:

function Output-FolderData($Mailbox, $Folder) {
        # Produce the output string
        $Output = New-Object System.Collections.ArrayList
        [void]$Output.add("User=`"$($Mailbox.SamAccountName)`"")
        [void]$Output.add("Folder=`"$($Folder.FolderPath)`"")
        [void]$Output.add("Type=`"$($Folder.FolderType)`"")
        [void]$Output.add("Size=$($Folder.FolderSize.ToBytes())")
        [void]$Output.add("Items=$($Folder.ItemsInFolder)")

        [string]::join(" ", $Output)
}

[jeff]

Unfortunately no... I just installed the Exchange app last week from a fresh download and used the TAs (for Exchange 2010) that were included on the forwarders. I verified from a fresh download of 1.1.4 that my TAs match what was included. All forwarders are Universal Forwarders sending to our central indexers. The get-mailboxstats.ps1, for instance, doesn't appear to include a timestamp.