topic Re: data stop getting indexed for couple of hours in Getting Data In

data stop getting indexed for couple of hours

Paul1896 — Tue, 12 Feb 2019 13:11:51 GMT

Hello,

we have problems with some log files which are randomly don't get indexed for a couple of hours. There is no log rotation during this time period and sometimes even no restart of the splunk forwarder is neccessary to start again with indexing.

Output splunkd.log filtered for affected "audit_log":

 02-12-2019 12:09:57.358 +0100 DEBUG ChunkedLBProcessor - Chunked Line Breaker Processing has been disabled for for sourcetype::audit_log
 02-12-2019 12:09:57.358 +0100 INFO  UTF8Processor - Converting using CHARSET="UTF-8" for conf "source::/xxx/audit.log|host::xxx|Haudit_log|339419"
02-12-2019 06:33:16.783 +0100 INFO  S2SSender - Abandoning channel with code=2, conf="source::/xxx/audit.log|host::xxx|audit_log|339419", unique_id=422585, last_touched=1549948674, last_touched_asctime="Tue Feb 12 06:17:54 2019", age=922281, ttl=300000
02-12-2019 06:17:54.985 +0100 INFO  Metrics - group=per_sourcetype_thruput, series="audit_log", kbps=0.06303521503133032, eps=0.5483843194729626, kb=1.9541015625, ev=17, avg_age=0.6470588235294118, max_age=5
02-12-2019 06:17:54.503 +0100 DEBUG TcpOutputProc - Pushed eventId=61 on chanId=422585 to back of tcp client (tcp output) queue. source:source::/xxx/audit.log|host::xxx|audit_log|339419

Re: data stop getting indexed for couple of hours

woodcock — Tue, 12 Feb 2019 16:22:36 GMT

If there is log rotation and you do not have some kind of housekeeping setup to delete the older files, the Splunk forwarder will get slower and slower and slower. Once you hit thousands of files in the same directory, Splunk will seem to stop forwarding completely. You have to keep the number of dead files low.

Re: data stop getting indexed for couple of hours

Paul1896 — Wed, 13 Feb 2019 07:54:33 GMT

Hey @woodcock thanks for your reply! We have a housekeeping in place and the monitor doesn't match the older logs which are already rotated. So in my understanding the size & total of files of/in the folder itself isn't important in case if your monitor just indexing a specified logfile in it. Please correct me if I'm wrong. We also facing the problem only from time to time and after a while or a reboot the logging works without any problems.

Re: data stop getting indexed for couple of hours

woodcock — Wed, 13 Feb 2019 16:08:15 GMT

Your understanding is completely wrong. Even though the rotated files do not match the monitor stanza, they still have a deadly impact on the forwarder. Splunk still has to sort through them and when they pile up, Splunk performance will crater. Restarting Splunk causes it to work for a short short spurt and then it goes right back to poor performance. Here is an easy way to fix it.

https://answers.splunk.com/answers/309910/how-to-monitor-a-folder-for-newest-files-only-file.html

Re: data stop getting indexed for couple of hours

woodcock — Wed, 13 Feb 2019 17:26:23 GMT

What version of Splunk are you using? There are some v6 releases that have big problems like this:

https://answers.splunk.com/answers/549663/splunk-661-stops-monitoring-files.html#answer-718797

Re: data stop getting indexed for couple of hours

ddrillic — Wed, 13 Feb 2019 18:12:33 GMT

@Paul1896 - about how many files we are talking about on this path?

Re: data stop getting indexed for couple of hours

Paul1896 — Thu, 14 Feb 2019 13:35:08 GMT

@ddrillic We're talking just about 220 files

Re: data stop getting indexed for couple of hours

Paul1896 — Thu, 14 Feb 2019 13:35:23 GMT

We're using Splunk 7.1.5

Re: data stop getting indexed for couple of hours

ddrillic — Thu, 14 Feb 2019 15:09:26 GMT

If that's the case, the number of files couldn't be the issue.

Re: data stop getting indexed for couple of hours

woodcock — Thu, 14 Feb 2019 21:08:43 GMT

It can if the depth is not limited. Are there any ... in the file path? Are there hundreds of other potential path points with many files in them? This can still be the problem!!!

Re: data stop getting indexed for couple of hours

Paul1896 — Mon, 18 Feb 2019 08:58:39 GMT

Probably we've figured out the root cause. Last week we indexed 20 gb for this instance within one day and have a configured thruput of 256 kb. So it was only a delayed indexing and not a "outage" of the forwarder itself.

Re: data stop getting indexed for couple of hours

woodcock — Wed, 06 Mar 2019 09:16:04 GMT

That would definitely do it. Please click Accept here to close the question.