Microsoft Office 365 Reporting Add-on for Splunk: Why are the message time and Splunk query time different?

[davidblizzard]

Good morning all.

I am getting message trace data using this Add-on, however the times are off. The result time shows the time that the data was received, not the actual date/time the message was received. Can anyone help me get them matched up? This is the only app with this issue.

Thanks.

[jconger]

You may need top copy the props.conf from the add-on to your search head(s). Here are the relevant time-related settings in props.conf from the add-on:

[ms:o365:reporting:messagetrace]
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TIME_PREFIX = "Received": "

Does the TIME_FORMAT match what you are seeing in your data?

[davidblizzard]

Thanks for the response. This is a single Windows server installation, so all components are on the same server. Here are my props file entries you are referring to:

[ms:o365:reporting:messagetrace]
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = 0
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
TIME_PREFIX = "DateReceived":

I changed the lookahead to 19 while troubleshooting based on similar issues reported (it was initially 30). I would have no issue setting that back.

The only other difference i see here is in the time_format, you show a % between S and Z. Mine is not separated by a %.

[jconger]

The TIME_PREFIX should be "Received": "
The MAX_TIMESTAMP_LOOKAHEAD is too short also. The timestamps in the data can be 27 characters long.

[davidblizzard]

Thank you. I have upgraded to the latest version and configured everything appropriately. Things are going well. The only issue I'm having now is the timezone. The query time matches the UTC time in the trace. I'd like it to display Central timezone. I've tried adding TZ = US/CENTRAL as well as TZ = America/Chicago to the props file, still not working.

[medunmeyer]

Try using TZ=UTC. That matches the TZ of the O365MessageTrace logs. Then they will display at US/Central if your servers/clients are set for that TimeZone.

[davidblizzard]

Good morning medunmeyer. That is not working either. More info below. Any help would be appreciated. Thanks.

My props file is below. I've tried setting TZ = UTC, America/Chicago, US/Central to no avail.
I have also removed the %Z from the Time_Format line thinking it may essentially disable the TZ statement.

The 'Received' data from the trace looks like this:

Received: 2018-10-24T09:33:56.8119313

Splunk Query time looks like this:

10/24/18
9:33:56.000 AM

Should be:

10/24/18
4:33:56.000 AM

[ms:o365:reporting:messagetrace]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = 0
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TZ = UTC
TIME_PREFIX = "Received": "
category = Splunk App Add-on Builder
pulldown_type = 1
EVAL-vendor_product = Microsoft Office 365
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_1 = RecipientAddress AS recipient
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_10 = SenderAddress AS orig_src
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_11 = SenderAddress AS src_user
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_12 = ToIP AS dest
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_13 = FromIP AS src
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_4 = MessageId AS message_id
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_5 = Subject AS subject
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_6 = Size AS size
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_7 = MessageTraceId AS internal_message_id
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_8 = Status AS action
FIELDALIAS-aob_gen_ms_o365_reporting_messagetrace_alias_9 = RecipientAddress AS orig_recipient

[ms:o365:reporting:mailtraffic]
SHOULD_LINEMERGE = 0
category = Splunk App Add-on Builder
pulldown_type = 1