Getting Data In

Are events from a particular source not getting indexed every night from 8pm to 12am, but events show up 24 hour later?

mjkenney
Explorer

Events from a particular source do not get indexed every night from 8pm to 12am. However 24 hours later the events show up. What is happening to the events? Is this an issue with Splunk or the system sending the logs. The source is generating and sending logs during this time.

0 Karma
1 Solution

mjkenney
Explorer

Figured it out. Added following to props.conf for host

TIME_PREFIX = time="
TZ = UTC

Fixed the problem. This is for Sonicwall logs btw.

View solution in original post

0 Karma

mjkenney
Explorer

Figured it out. Added following to props.conf for host

TIME_PREFIX = time="
TZ = UTC

Fixed the problem. This is for Sonicwall logs btw.

0 Karma

twinspop
Influencer

Without more info, I'm going to bet it's a timezone issue. Or possibly your servers' time is off? You can check your events' timestamps vs index time by comparing the _time and _indextime fields.

<some search> | eval diff=_time-_indextime | stats max(diff) as max min(diff) as min by host source

Anything more than a few seconds of difference should be concerning. Most often I see multihour differences skewing to the future when a server is logging in UTC, but Splunk is interpreting the time as local (EDT or similar).

mjkenney
Explorer

Figured it out. Added following to props.conf for host

TIME_PREFIX = time="
TZ = UTC

Fixed the problem. This is for Sonicwall logs btw.

0 Karma

twinspop
Influencer

I'm happy you found a solution. Be careful, by setting the TZ at host level, you are saying that any log that comes from there will be UTC. That very well could be accurate -- just remember it. 🙂

mjkenney
Explorer

It's definitely a timezone issue.

Event from September 9th, 7:59:59 PM, shows the UTC date/time stamp is going to flip to Sept 10th
Splunk Time
9/9/15 7:59:59.000 PM

Event
Sep 9 19:59:59 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-09 23:59:59 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1779272 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112

Event from September 9th, 8:00:00 PM, shows the UTC date/time stamp is Sept 11th and not the 10th.
Splunk Time
9/9/15 8:00:00.000 PM

Event
Sep 10 20:00:00 4.4.4.4 id=firewall sn=ABCDEFG time="2015-09-11 00:00:00 UTC" fw=123.456.78.9 pri=1 c=512 m=522 msg="Malformed or unhandled IP packet dropped" sess=None n=1865689 src=1.1.1.1:0:X0 dst=2.2.2.2:0:X0 proto=112

What do I need to adjust here to correct this?

0 Karma

twinspop
Influencer

If all your logs of this type are logging in UTC, you should set the TZ option in a props.conf file on the indexer(s), using the sourcetype as the stanza opening. Example

[yoursourcetype]
TZ = GMT

If you have an indexer app, put it in props.conf in that app bundle. Otherwise, you can put in $splunkhome/etc/system/local/props.conf. You'll need to restart the indexer after the change.

0 Karma

somesoni2
Revered Legend

The logging is done on the UTC time zone. And your Splunk timezone is EDT. What is expected here? Do you want to show "2015-09-11 00:00:00 UTC" to show up the Splunk Time as "2015-09-11 00:00:00 EDT" ??

0 Karma

mjkenney
Explorer

Why does it jump an entire day? Shouldn't it read 2015-09-10 00:00:00 UTC, not 09-11?

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...