Are events from a particular source not getting indexed every night from 8pm to 12am, but events show up 24 hour later?

[mjkenney]

Events from a particular source do not get indexed every night from 8pm to 12am. However 24 hours later the events show up. What is happening to the events? Is this an issue with Splunk or the system sending the logs. The source is generating and sending logs during this time.

[mjkenney · Accepted solution]

Figured it out. Added following to props.conf for host

TIME_PREFIX = time="
TZ = UTC

Fixed the problem. This is for Sonicwall logs btw.

[twinspop]

Without more info, I'm going to bet it's a timezone issue. Or possibly your servers' time is off? You can check your events' timestamps vs index time by comparing the _time and _indextime fields.

<some search> | eval diff=_time-_indextime | stats max(diff) as max min(diff) as min by host source

Anything more than a few seconds of difference should be concerning. Most often I see multihour differences skewing to the future when a server is logging in UTC, but Splunk is interpreting the time as local (EDT or similar).

[mjkenney]

Figured it out. Added following to props.conf for host

TIME_PREFIX = time="
TZ = UTC

Fixed the problem. This is for Sonicwall logs btw.

[twinspop]

I'm happy you found a solution. Be careful, by setting the TZ at host level, you are saying that any log that comes from there will be UTC. That very well could be accurate -- just remember it. 🙂

[mjkenney]

It's definitely a timezone issue.

Event from September 9th, 7:59:59 PM, shows the UTC date/time stamp is going to flip to Sept 10th
Splunk Time
9/9/15 7:59:59.000 PM

Event
Sep 9 19:59:59 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-09 23:59:59 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1779272 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112

Event from September 9th, 8:00:00 PM, shows the UTC date/time stamp is Sept 11th and not the 10th.
Splunk Time
9/9/15 8:00:00.000 PM

Event
Sep 10 20:00:00 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-11 00:00:00 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1865689 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112

What do I need to adjust here to correct this?

[twinspop]

If all your logs of this type are logging in UTC, you should set the TZ option in a props.conf file on the indexer(s), using the sourcetype as the stanza opening. Example

[yoursourcetype]
TZ = GMT

If you have an indexer app, put it in props.conf in that app bundle. Otherwise, you can put in $splunkhome/etc/system/local/props.conf. You'll need to restart the indexer after the change.

[somesoni2]

The logging is done on the UTC time zone. And your Splunk timezone is EDT. What is expected here? Do you want to show "2015-09-11 00:00:00 UTC" to show up the Splunk Time as "2015-09-11 00:00:00 EDT" ??

[mjkenney]

Why does it jump an entire day? Shouldn't it read 2015-09-10 00:00:00 UTC, not 09-11?