topic Re: Log files not being indexed in Splunk Enterprise

Log files not being indexed

YanwuGuTelus — Mon, 06 Mar 2017 00:33:06 GMT

I have log files that are not being indexed by Splunk, and don't show up in the search. In the same folder, other files are being indexed fine. The only difference I could find between these files is that the ones that are not being indexed have some tab and new line characters, i.e. \t, \n, \r

Re: Log files not being indexed

woodcock — Mon, 06 Mar 2017 05:03:34 GMT

Try adding this to props.conf:

NO_BINARY_CHECK = true

Re: Log files not being indexed

YanwuGuTelus — Mon, 06 Mar 2017 18:17:02 GMT

I have tried it - add the line, waited for 10 mins. Not solving the issue.
Do I need reboot my server?

Re: Log files not being indexed

mattymo — Tue, 07 Mar 2017 00:59:29 GMT

To determine why files may not be ingested, try one of the following:

1- Check the inputstatus command on the Splunk instance that will be ingesting the file

[splunker@n00bserver bin]$ ./splunk list inputstatus

You may want to send the output to a file as the output can be large. If the file was read it will look something like this:

/home/splunker/splunk/var/log/splunk/splunk.log
        file position = 503
        file size = 503
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = finished reading

If ignored it may look something like this:

    /home/splunker/splunk/etc/apps/SA-Utils/appserver/modules/SOLNTreeNav/SOLNTreeNav.css
        parent = $SPLUNK_HOME/etc/*apps/....css*
        type = ignored file (crc conflict, needs crcSalt)

2- Check Splunk internal logs

try searching:

index=_internal  TailReader ERROR OR WARN

index=_internal /path/to/your/file.log

Re: Log files not being indexed

woodcock — Tue, 07 Mar 2017 21:12:41 GMT

You need to restart splunk, yes.

Re: Log files not being indexed

YanwuGuTelus — Tue, 07 Mar 2017 22:17:05 GMT

Ya, I restarted, still not being indexed.

Re: Log files not being indexed

YanwuGuTelus — Wed, 08 Mar 2017 00:23:43 GMT

So I found various unprocessed files logged with "type = ignored file (crc conflict, needs crcSalt)". What to do in this situation?

I also found some files logged with "type = finished reading" but their contents are still not showing in Splunk search.

Which log file should I search further in? I see a bunch of files under $SPLUNK_HOME/var/log/splunk/, is that the right place to look?

Re: Log files not being indexed

mattymo — Wed, 08 Mar 2017 05:16:00 GMT

When Splunk finds a file it reads the first 256 bytes of the file and computes a hash as a way of determining whether we have already read the file.

See here:

https://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Howlogfilerotationishandled

In your case the file may have large headers or a common preamble in the first 256 bytes that causes Splunk to believe it has seen the file already.

You should examine the file and if indeed this is a unique file you can add crcsalt to the inputs.conf for this file to have splunk add the path to the crc check as a way to ensure Splunk knows this is a different file...See monitor syntax and examples here:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Monitorfilesanddirectorieswithinputs.conf

It is also possible to tell Splunk to extend the check further than 256 so it can reach the unique data...see crcInitLength in the first link i provided.

You will want to be careful with these settings if you are using wildcards in yout directory monitors...generally you only want to apply explicitly to certain files to avoid duplicate ingestion

As for the files that are finished reading yet arent in Splunk, have you searched All time for the source path?

As for the logs, yes that is a valuable directory with many useful logs. Splunk ingests many of them by default so that you can search them from the gui, you can see much of what the commands I provided showed us by searching index=_internal sourcetype=splunkd tailreader but at this point we just need to analyze the files failing crc check and adjust the inputs and figure out where the data from the finished files are

Re: Log files not being indexed

mattymo — Wed, 08 Mar 2017 05:17:14 GMT

You will want to roll that setting back. Better to find root cause before altering settings.

Re: Log files not being indexed

buckiboy — Fri, 15 Nov 2019 12:02:37 GMT

Hi, have you managed to resolve this? I am having the same problem with logs being shown as indexed, licence used for them but unable to actually see them in Splunk.

Re: Log files not being indexed

woodcock — Mon, 18 Nov 2019 04:07:14 GMT

Are these in the name of the files or the contents?

Re: Log files not being indexed

woodcock — Mon, 18 Nov 2019 04:12:00 GMT

There are many reasons for this including:

Files are presumed to be already forwarded; this is the type = ignored file (crc conflict, needs crcSalt) problem and the solution is to add this to the inputs.conf on the UF:

crcSalt = <SOURCE>

It could be that the timestamps are too old (change MAX_DAYS_AGO) or that they are being thrown into the future by accident so they are not showing up in your search. To test for this, set your Time picker to Advanced and earliest to 0 and latest to @d+10y and use _index_earliest=-30d _index_latest=now.