Getting Data In

Why is there large data latency coming from syslog?

perfecto25
Path Finder

Hello, we have a proxy network appliance running Websense, sending its logs via syslog to Splunk,

We have a data latency alert configured to alert if latency is large,

search $search_args$ _index_earliest=-1d@d _index_latest=@d 
 | eval lag_sec = (_indextime-_time)
 | eval lag_hrs = lag_sec/(60*60)
 | eval delay_hrs  = if( lag_hrs > 0.5,   lag_hrs, "")
 | eval future_sec = if( lag_sec < -1, -1*lag_sec, "")
 | eval containsGap = if(delay_hrs!="" OR future_sec!="", "true", "false")
 | stats max(delay_hrs),
         max(future_sec),
         count(eval(containsGap="true")) as countGaps,
         count(_raw) as countEvents,
         by splunk_server index host sourcetype source
 | eval pecentGaps = countGaps / countEvents*100
 | where pecentGaps>5 
 | sort host, sourcetype, source

We started to get large latency (2 hour (7200 seconds) gap between received events timestamp and when theyre indexed) in last few days, and I am trying to determine whats causing this,

alt text

We dont have a forwarder on this network device, and we arent seeing any additional network bottlenecks or traffic. Where can I look to troubleshoot data integrity latency?

Thanks

0 Karma

woodcock
Esteemed Legend

This is almost always due to incorrect interpretation of TimeZones (usually because there are no TZ values in the timestamps and there is no TZ= in any props.conf so each indexer uses the TZ value of its host OS (which shouldn't be, but might be, different on each indexer).

0 Karma

perfecto25
Path Finder

I checked the indexer, it has the host configured with the right TZ

[root@cgysplunk01 /opt/splunk]# cat ./etc/system/local/props.conf
[host::cgyxxpwcg02.xxxx]
TZ = America/Edmonton

The indexer itself is EST TZ

[root@cgysplunk01 /opt/splunk]# cat /etc/sysconfig/clock
ZONE="America/New_York"

0 Karma

xpac
SplunkTrust
SplunkTrust

Can you please show an example event?

0 Karma

ddrillic
Ultra Champion
0 Karma

xpac
SplunkTrust
SplunkTrust

Latency is always 7199 seconds? This sounds more like an issue with a wrong timezone than actual latency...

0 Karma

perfecto25
Path Finder

no, latency varies but all are above -7000s

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...