Splunk Search

_time and date_hour don't match


Yes, I have already checked my user time zone setting. My TZ setting and all my involved servers, forwarder and Splunk servers, are all configured for the same TZ.

I have two servers that are configured the same and have the same use case. Server A is sending events where the _time and date_hour are differing in search. The hour of the timestamp in the log that we are consuming is matching date_hour.

Server B is sending events where the _time, date_hour, and the hour of the timestamp in the log match.

I am performing the search at the same time and other users are seeing the same results (and are asking me why there is a variance).

I have confirmed that both servers are using the same deployed apps. And Server A was working this past Sunday, but no changes were made to the Splunk configuration for these servers between then and Tuesday when the incorrect _time appeared. Both servers in this example are monitoring the same log, its just specific to their own server.

Any ideas?

0 Karma


Hello @jasonwagner

  • can you verifiy the local time on both severs?
  • additionally check for time drift in the syslog/messages/journal of the particular server: index=_internal host=serverB ntp* OR adjust
0 Karma


Thank you, @PaveIP. I verified again that Server A and Server B both have the same time and time zone. If they were different, we would have other application issues besides Splunk.

I also performed the index=_internal host=serverB ntp* OR adjust search for the past 30 days against both Server A and Server B and received no results.

0 Karma

Ultra Champion

date_hour is default field at search time.
but it is not reliable.

0 Karma


The problem I'm experiencing is that _time is the field that is unreliable here, not date_hour.

0 Karma

Ultra Champion

can you share a data sample and your props.conf for that particular sourcetype?

0 Karma


Here you go, I have to obfuscate some of the event date:

SEDCMD-remove = s/(\s{3}at.)|(\n\s+---\s.)|(\nServer stack trace:)|(\nException\s\w+\s.*:)//g
SEDCMD-spaces = s/[\n\r]+//g
SEDCMD-nullblock = s/(XXX(Xxxxx=(null):(null)\,PosId=(null):(null)\,IP=(null))\ InitialTrans(ID=(null)\,\ SEQ=(null)))/SED-nb/g
TRANSFORMS-Combined = transform_ak_f2p,transform_ak_ce,transform_ak_all
TRANSFORMS-Type_Fields = extract_type
TRANSFORMS-Level_Fields = extract_level
TRANSFORMS-Message_Fields = extract_message

2020-05-12 13:05:47,817 [Upload7] ERROR (xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.xxxx.FallbackMessageXmlQueueHandler) SED-nb Failed to persists message to xxxx xxxx saving 1 Message Xmls to DB. Error Message: The xxxx operation was interrupted: xxxx close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

_time = 2020-05-12T08:05:47.817-05:00
date_hour = 13

date_mday = 12

date_minute = 5
date_month = may

date_second = 47

date_wday = tuesday
date_year = 2020

date_zone = local

0 Karma
Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...