universal forwarder delay - 8 minutes

[rewritex]

Any ideas why I am seeing an 8 minute delay in the UF -> Index data?

The UF is monitoring a logfile that is consistently generating realtime data. When I view the index from my SH, I am unable to see anything before 8 minutes. I am currently using a cluster: 1SH, (1indx+1indx) RF2 SF2 cluster, Master, UF w/ index autodiscover set ... And it seems all other index are reacting in the same way so I am assuming this is a global setting somewhere. I'm currently reading up on limits.conf and pipeline/parallel parellization.

Thank You,
Sean

[twinspop · Accepted solution]

I would search _internal for indications that pipelines are blocked or paused. In all likelihood, your indexer is not up to the indexing load being presented to it. Either slow disk or not enough CPU. If you have many cores, you could try adding pipelines. Otherwise, an additional indexer may be your answer. (1 extra pipeline will result in 10-12 CPUs being used, total, for indexing. Don't add a parallel pipeline unless you have many more cores than that still available for searching.)

[somesoni2]

Here are some troubleshooting steps for the same
see this first
http://docs.splunk.com/Documentation/Splunk/6.3.3/Troubleshooting/Troubleshootingeventsindexingdelay

https://answers.splunk.com/answers/39211/why-am-i-seeing-indexing-lag.html

https://answers.splunk.com/answers/247513/what-is-the-source-of-indexing-lag-and-how-to-fix.html