_time and date_hour don't match

[jasonwagner]

Yes, I have already checked my user time zone setting. My TZ setting and all my involved servers, forwarder and Splunk servers, are all configured for the same TZ.

I have two servers that are configured the same and have the same use case. Server A is sending events where the _time and date_hour are differing in search. The hour of the timestamp in the log that we are consuming is matching date_hour.

Server B is sending events where the _time, date_hour, and the hour of the timestamp in the log match.

I am performing the search at the same time and other users are seeing the same results (and are asking me why there is a variance).

I have confirmed that both servers are using the same deployed apps. And Server A was working this past Sunday, but no changes were made to the Splunk configuration for these servers between then and Tuesday when the incorrect _time appeared. Both servers in this example are monitoring the same log, its just specific to their own server.

Any ideas?

[PavelP]

Hello @jasonwagner

• can you verifiy the local time on both severs?
• additionally check for time drift in the syslog/messages/journal of the particular server: index=_internal host=serverB ntp* OR adjust

[jasonwagner]

Thank you, @PaveIP. I verified again that Server A and Server B both have the same time and time zone. If they were different, we would have other application issues besides Splunk.

I also performed the index=_internal host=serverB ntp* OR adjust search for the past 30 days against both Server A and Server B and received no results.

[to4kawa]

date_hour is default field at search time.
but it is not reliable.

[jasonwagner]

The problem I'm experiencing is that _time is the field that is unreliable here, not date_hour.

[adonio]

can you share a data sample and your props.conf for that particular sourcetype?

[jasonwagner]

Here you go, I have to obfuscate some of the event date:

[props_stanza_in_question]
CHARSET = UTF-8
MAX_TIMESTAMP_LOOKAHEAD = 23
FIELDALIAS-IP = IP AS src_ip
SEDCMD-remove = s/(\s{3}at.)|(\n\s+---\s.)|(\nServer stack trace:)|(\nException\s\w+\s.*:)//g
SEDCMD-spaces = s/[\n\r]+//g
SEDCMD-nullblock = s/(XXX(Xxxxx=(null):(null)\,PosId=(null):(null)\,IP=(null))\ InitialTrans(ID=(null)\,\ SEQ=(null)))/SED-nb/g
TRANSFORMS-Combined = transform_ak_f2p,transform_ak_ce,transform_ak_all
TRANSFORMS-Type_Fields = extract_type
TRANSFORMS-Level_Fields = extract_level
TRANSFORMS-Message_Fields = extract_message

Event:
2020-05-12 13:05:47,817 [Upload7] ERROR (xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.FallbackMessageXmlQueueHandler) SED-nb Failed to persists message to xxxx xxxx saving 1 Message Xmls to DB. Error Message: The xxxx operation was interrupted: xxxx close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Fields:
_time = 2020-05-12T08:05:47.817-05:00
date_hour = 13

date_mday = 12

date_minute = 5
date_month = may

date_second = 47

date_wday = tuesday
date_year = 2020

date_zone = local