Yes, I have already checked my user time zone setting. My TZ setting and all my involved servers, forwarder and Splunk servers, are all configured for the same TZ.
I have two servers that are configured the same and have the same use case. Server A is sending events where the _time and date_hour are differing in search. The hour of the timestamp in the log that we are consuming is matching date_hour.
Server B is sending events where the _time, date_hour, and the hour of the timestamp in the log match.
I am performing the search at the same time and other users are seeing the same results (and are asking me why there is a variance).
I have confirmed that both servers are using the same deployed apps. And Server A was working this past Sunday, but no changes were made to the Splunk configuration for these servers between then and Tuesday when the incorrect _time appeared. Both servers in this example are monitoring the same log, its just specific to their own server.
Any ideas?
Hello @jasonwagner
index=_internal host=serverB ntp* OR adjust
Thank you, @PaveIP. I verified again that Server A and Server B both have the same time and time zone. If they were different, we would have other application issues besides Splunk.
I also performed the index=_internal host=serverB ntp* OR adjust
search for the past 30 days against both Server A and Server B and received no results.
date_hour is default field at search time.
but it is not reliable.
The problem I'm experiencing is that _time is the field that is unreliable here, not date_hour.
can you share a data sample and your props.conf for that particular sourcetype?
Here you go, I have to obfuscate some of the event date:
[props_stanza_in_question]
CHARSET = UTF-8
MAX_TIMESTAMP_LOOKAHEAD = 23
FIELDALIAS-IP = IP AS src_ip
SEDCMD-remove = s/(\s{3}at.)|(\n\s+---\s.)|(\nServer stack trace:)|(\nException\s\w+\s.*:)//g
SEDCMD-spaces = s/[\n\r]+//g
SEDCMD-nullblock = s/(XXX(Xxxxx=(null):(null)\,PosId=(null):(null)\,IP=(null))\ InitialTrans(ID=(null)\,\ SEQ=(null)))/SED-nb/g
TRANSFORMS-Combined = transform_ak_f2p,transform_ak_ce,transform_ak_all
TRANSFORMS-Type_Fields = extract_type
TRANSFORMS-Level_Fields = extract_level
TRANSFORMS-Message_Fields = extract_message
Event:
2020-05-12 13:05:47,817 [Upload7] ERROR (xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.FallbackMessageXmlQueueHandler) SED-nb Failed to persists message to xxxx xxxx saving 1 Message Xmls to DB. Error Message: The xxxx operation was interrupted: xxxx close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Fields:
_time = 2020-05-12T08:05:47.817-05:00
date_hour = 13
date_mday = 12
date_minute = 5
date_month = may
date_second = 47
date_wday = tuesday
date_year = 2020
date_zone = local