Splunk not indexing some files

[cafissimo]

Hello, I am trying to index some files from a lightforwarder on a unix directory (HP-UX), by writing a proper inputs.conf file. Splunk send to the indexer only some files and seems to ignore others and I am sure that inputs.conf is correct. Splunk version is 4.1.3 Has anyone experienced a similar problem? Luca.

I am getting this messages in the indexer (internal index) about the file I am trying to index (file that is always growing on the forwarder)

10-06-2010 15:23:49.182 DEBUG TailingProcessor - Deferred notification for path='/xxx/logs/weblogic/ib8o0/banking-jsp.log'.

10-06-2010 15:23:49.182 DEBUG TailingProcessor - Have seen this item before.

10-06-2010 15:23:49.182 DEBUG TailingProcessor - Will attempt to read file: /xxx/logs/weblogic/ib8o0/banking-jsp.log from existing fd.

10-06-2010 15:23:49.183 DEBUG TailingProcessor - About to read data (Reusing existing fd for file='/xxx/logs/weblogic/ib8o0/banking-jsp.log').

10-06-2010 15:23:49.183 DEBUG TailingProcessor - Hit EOF immediately.

10-06-2010 15:23:49.183 DEBUG TailingProcessor - Have definitely hit EOF.

[Lowell · Accepted solution]

You'll probably need to provide more details about what you are trying to do. You inputs.conf stanza would help. Specific file names that are being ignored would help. (Click the "edit" link under your question to add additional details to your post)

In the mean time, I would recommend seeing if you have any source patterns setup to block some of your files. You can run the following command on some of your files that are currently being blocked:

$SPLUNK_HOME/bin/splunk test sourcetype /path/to/your/logfiles/missing.log

You may also want to run this a the "splunk" user, just to eliminate any possibility of file permission issues.

It may also be enlightening to look around in the _internal index to see if any messages about not being able to process the files in question. (This of course assumes you are forwarding your _internal index). Try a search like this:

 index=_internal sourcetype=splunkd /path/to/your/logfiles/*

[cafissimo · Accepted solution]

It seems it was a timestamp recognition problem. I have fixed TIME_FORMAT in props.conf and now the file is being indexed.

[Lowell · Accepted solution]

You'll probably need to provide more details about what you are trying to do. You inputs.conf stanza would help. Specific file names that are being ignored would help. (Click the "edit" link under your question to add additional details to your post)

In the mean time, I would recommend seeing if you have any source patterns setup to block some of your files. You can run the following command on some of your files that are currently being blocked:

$SPLUNK_HOME/bin/splunk test sourcetype /path/to/your/logfiles/missing.log

You may also want to run this a the "splunk" user, just to eliminate any possibility of file permission issues.

It may also be enlightening to look around in the _internal index to see if any messages about not being able to process the files in question. (This of course assumes you are forwarding your _internal index). Try a search like this:

 index=_internal sourcetype=splunkd /path/to/your/logfiles/*