Getting Data In

After indexing slowed down due to available space, how can I speed up Splunk forwarding and indexing to catch up?

Path Finder

There was a problem with available space for indexing and it had slowed down. But once the problem was fixed, the indexed events are still 7 hours older than the recent event in the source file (a application log file). Since the log file is being continuously written, I wonder if splunk will ever be able to catchup:

a) Is there any way, I can speed up forwarding (universal forwarder) and indexing so that splunk can catch up quickly. Since the log file is going to be used for real time monitoring of mission critical application, we cannot afford to take many hours for splunk to catch up after forwarder goes down or when indexing runs across a problem.

Tags (3)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

universal forwarder limits itself to 256Kbps, if your indexers can keep up, you can start to increase the limits.conf on the forwarder.

see this article
http://docs.splunk.com/Documentation/Splunk/6.2.2/Troubleshooting/Troubleshootingeventsindexingdelay...

View solution in original post

Path Finder

I am still confused why my last 5 minutes in realtime show 7 hour old events while realtime(all time) shows most recent event. I executed following search for the two different time windows

Search

index=*410* sourcetype="dplogskv" source=/logs/messages host=abcd1202 | eval delay_sec=_indextime-_time | eval time = strftime(_time,"%Y-%m-%d %T-%z") | eval indextime = strftime(_indextime,"%Y-%m-%d %T-%z") | eval now=strftime(now(),"%Y-%m-%d %T-%z") | table time indextime now delay_sec TxnId date_zone source sourcetype host

5 minute (realtime)

time indextime now delay_sec TxnId date_zone
2015-03-24 11:07:20--0600 2015-03-24 04:07:18--0600 2015-03-24 11:11:23--0600 -25202 42280897 local

2015-03-24 11:07:20--0600 2015-03-24 04:07:18--0600 2015-03-24 11:11:23--0600 -25202 36151977 local

2015-03-24 11:07:20--0600 2015-03-24 04:07:18--0600 2015-03-24 11:11:23--0600 -25202 40497509 local

2015-03-24 11:07:26--0600 2015-03-24 04:07:24--0600 2015-03-24 11:11:23--0600 -25202 15209107 local

2015-03-24 11:07:29--0600 2015-03-24 04:07:27--0600 2015-03-24 11:11:23--0600 -25202 15209187 local

2015-03-24 11:07:56--0600 2015-03-24 04:07:54--0600 2015-03-24 11:11:23--0600 -25202 42305713 local

alltime (realtime)
time indextime now delay_sec date_zone
2015-03-24 18:13:34--0600 2015-03-24 11:13:34--0600 2015-03-24 11:12:51--0600 -25200 local

0 Karma

Splunk Employee
Splunk Employee

universal forwarder limits itself to 256Kbps, if your indexers can keep up, you can start to increase the limits.conf on the forwarder.

see this article
http://docs.splunk.com/Documentation/Splunk/6.2.2/Troubleshooting/Troubleshootingeventsindexingdelay...

View solution in original post

Path Finder

Thank you yannK. I went through the link and followed the Timezone discrepancy section . Through that I found that it was a false alert. Actually the latest events are getting indexed but I was misled by the outputs when I use search time as last 5 minutes (realtime) v/s all time (real time). I still have an issue detailed below (as a 2nd answer due to length of message) , let me know if it needs to be a separate question.

0 Karma

Path Finder

After spending lot of hours and going through other community questions I figured that indexer was not recognizing the TZ attribute when under SourceType Stanza and needs to be specified under Source stanza. Weird part is that I have exactly same setup where the same log file is being forwarded to another indexer pool and there the TZ attribute is working when under SourceType stanza. It seems like some kind of defect in 6.1.3

0 Karma