I have log files that are not being indexed by Splunk, and don't show up in the search. In the same folder, other files are being indexed fine. The only difference I could find between these files is that the ones that are not being indexed have some tab and new line characters, i.e. \t, \n, \r
There are many reasons for this including:
Files are presumed to be already forwarded; this is the type = ignored file (crc conflict, needs crcSalt)
problem and the solution is to add this to the inputs.conf
on the UF:
crcSalt = <SOURCE>
It could be that the timestamps are too old (change MAX_DAYS_AGO
) or that they are being thrown into the future by accident so they are not showing up in your search. To test for this, set your Time picker
to Advanced
and earliest
to 0
and latest
to @d+10y
and use _index_earliest=-30d _index_latest=now
.
Are these in the name
of the files or the contents
?
Hi, have you managed to resolve this? I am having the same problem with logs being shown as indexed, licence used for them but unable to actually see them in Splunk.
To determine why files may not be ingested, try one of the following:
1- Check the inputstatus command on the Splunk instance that will be ingesting the file
[splunker@n00bserver bin]$ ./splunk list inputstatus
You may want to send the output to a file as the output can be large. If the file was read it will look something like this:
/home/splunker/splunk/var/log/splunk/splunk.log
file position = 503
file size = 503
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
If ignored it may look something like this:
/home/splunker/splunk/etc/apps/SA-Utils/appserver/modules/SOLNTreeNav/SOLNTreeNav.css
parent = $SPLUNK_HOME/etc/*apps/....css*
type = ignored file (crc conflict, needs crcSalt)
2- Check Splunk internal logs
try searching:
index=_internal TailReader ERROR OR WARN
OR
index=_internal /path/to/your/file.log
So I found various unprocessed files logged with "type = ignored file (crc conflict, needs crcSalt)". What to do in this situation?
I also found some files logged with "type = finished reading" but their contents are still not showing in Splunk search.
Which log file should I search further in? I see a bunch of files under $SPLUNK_HOME/var/log/splunk/, is that the right place to look?
When Splunk finds a file it reads the first 256 bytes of the file and computes a hash as a way of determining whether we have already read the file.
See here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Howlogfilerotationishandled
In your case the file may have large headers or a common preamble in the first 256 bytes that causes Splunk to believe it has seen the file already.
You should examine the file and if indeed this is a unique file you can add crcsalt to the inputs.conf for this file to have splunk add the path to the crc check as a way to ensure Splunk knows this is a different file...See monitor syntax and examples here:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Monitorfilesanddirectorieswithinputs.conf
It is also possible to tell Splunk to extend the check further than 256 so it can reach the unique data...see crcInitLength in the first link i provided.
You will want to be careful with these settings if you are using wildcards in yout directory monitors...generally you only want to apply explicitly to certain files to avoid duplicate ingestion
As for the files that are finished reading yet arent in Splunk, have you searched All time for the source path?
As for the logs, yes that is a valuable directory with many useful logs. Splunk ingests many of them by default so that you can search them from the gui, you can see much of what the commands I provided showed us by searching index=_internal sourcetype=splunkd tailreader
but at this point we just need to analyze the files failing crc check and adjust the inputs and figure out where the data from the finished files are
Try adding this to props.conf:
NO_BINARY_CHECK = true
I have tried it - add the line, waited for 10 mins. Not solving the issue.
Do I need reboot my server?
You need to restart splunk, yes.
Ya, I restarted, still not being indexed.
You will want to roll that setting back. Better to find root cause before altering settings.