topic Splunk stopped indexing some files in Getting Data In

Splunk stopped indexing some files

jaydee77ca — Thu, 08 Dec 2011 22:34:51 GMT

Hi,

We are indexing a directory on one of our servers (/sonic/logs) and Splunk suddenly stopped indexing a few of the files. It appears to have stopped after the log file was rolled over one night. I checked the TailingProcessor status via the REST URL (https://ourserver:8090/services/admin/inputstatus/TailingProcessor%3AFileStatus) and for the files it stopped indexing it says 100% and "finished reading". However I know that there are still new entries being written to the file that aren't showing up in Splunk.

What has caused Splunk to stop indexing these files and how can we get it to resume?

We are running Splunk 4.2.2 on AIX.

Thanks!

Re: Splunk stopped indexing some files

_d_ — Fri, 09 Dec 2011 00:30:03 GMT

Do the newly rolled log files inherit any headers from their predecessors? If yes, then you might need crcSalt=<SOURCE> in your input stanza. Check here for more details: http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Also, I would check $SPLUNK_HOME/var/log/splunk/splunkd.log for additional information/clues.

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: Splunk stopped indexing some files

jaydee77ca — Fri, 09 Dec 2011 22:52:23 GMT

Yeah, they will likely all have the same first line as it is usually:

[11/12/06 00:00:01] ID=AGENT (info) Log file rollover initiated...

Of course the timestamp is different.

The thing is, it's been indexing these files for months and we've never had any issues. Now all of a sudden it stopped. And actually it looks like it hasn't stopped completely as yesterday and the day before it indexed a half-dozen or so lines from the file but that's it. (Usually there are hundreds of lines or more)

I've tried restarting splunk on the server and that didn't seem to affect anything.

Re: Splunk stopped indexing some files

jaydee77ca — Fri, 09 Dec 2011 22:52:28 GMT

The documentation for crcSalt says that it shouldn't be used with rolling log files and these files are rolled so I'm not sure if that will help. 😕