Events from a particular source do not get indexed every night from 8pm to 12am. However 24 hours later the events show up. What is happening to the events? Is this an issue with Splunk or the system sending the logs. The source is generating and sending logs during this time.
Figured it out. Added following to props.conf for host
TIME_PREFIX = time="
TZ = UTC
Fixed the problem. This is for Sonicwall logs btw.
Figured it out. Added following to props.conf for host
TIME_PREFIX = time="
TZ = UTC
Fixed the problem. This is for Sonicwall logs btw.
Without more info, I'm going to bet it's a timezone issue. Or possibly your servers' time is off? You can check your events' timestamps vs index time by comparing the _time and _indextime fields.
<some search> | eval diff=_time-_indextime | stats max(diff) as max min(diff) as min by host source
Anything more than a few seconds of difference should be concerning. Most often I see multihour differences skewing to the future when a server is logging in UTC, but Splunk is interpreting the time as local (EDT or similar).
Figured it out. Added following to props.conf for host
TIME_PREFIX = time="
TZ = UTC
Fixed the problem. This is for Sonicwall logs btw.
I'm happy you found a solution. Be careful, by setting the TZ at host level, you are saying that any log that comes from there will be UTC. That very well could be accurate -- just remember it. 🙂
It's definitely a timezone issue.
Event from September 9th, 7:59:59 PM, shows the UTC date/time stamp is going to flip to Sept 10th
Splunk Time
9/9/15 7:59:59.000 PM
Event
Sep 9 19:59:59 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-09 23:59:59 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1779272 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112
Event from September 9th, 8:00:00 PM, shows the UTC date/time stamp is Sept 11th and not the 10th.
Splunk Time
9/9/15 8:00:00.000 PM
Event
Sep 10 20:00:00 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-11 00:00:00 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1865689 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112
What do I need to adjust here to correct this?
If all your logs of this type are logging in UTC, you should set the TZ option in a props.conf file on the indexer(s), using the sourcetype as the stanza opening. Example
[yoursourcetype]
TZ = GMT
If you have an indexer app, put it in props.conf in that app bundle. Otherwise, you can put in $splunkhome/etc/system/local/props.conf. You'll need to restart the indexer after the change.
The logging is done on the UTC time zone. And your Splunk timezone is EDT. What is expected here? Do you want to show "2015-09-11 00:00:00 UTC" to show up the Splunk Time as "2015-09-11 00:00:00 EDT" ??
Why does it jump an entire day? Shouldn't it read 2015-09-10 00:00:00 UTC, not 09-11?