Rolled log file not being indexed

[Jaci]

I monitor a log file (access_log) that gets rolled every night at 1 am using a copy command "cp /dev/null access_today". After the file rolls splunk randomly stops indexing the file until a day or two later after a roll. This only happens for this log file.

The log file is empty after the copy but usually has data within a minute or so. Is it possible that the Splunk database has an zero size CRC entry of this file? How can we check that?

I'm running splunk 4.0.9.

I have this set in my inputs.conf

[monitor:///host1/log/]  
crcSalt=<SOURCE>    
_whitelist = (access_log|error_log)

Here is a sample of the access_log file:

192.7.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 382777 - - - - 192.33.220.7 -
192.5.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 309851 - - - - 192.33.220.7 -
192.6.12.120 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 319522 - - - - 192.33.220.7 -
- - - [11/May/2010:01:35:34 -0400] 200 349 - - GET /xxx/xxx.html HTTP/0.9 754 - - - - 192.33.220.7 -
192.7.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 396867 - - - - 192.33.220.7 -

[Lowell]

I assume that your cp command it to essentially to truncate your log file because you don't want to restart your server? Any chance that you could use logrotate (with copytruncate mode) or use a pipe-based file rotator like rotatelogs (comes with apache I believe) instead?

[Lowell]

Out of curiosity, why do you have the crcSalt=<SOURCE> in there?

[Simeon · Accepted solution]

If you turn on the debug flags for file monitoring, Splunk will tell you the status of that file. You can review how to set these flags here:

http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs

Specifically, you should edit the $SPLUNK_HOME/etc/log.cfg file and set the following parameters:

category.FileInputTracker=DEBUG
category.selectProcessor=DEBUG
category.TailingProcessor=DEBUG

You will need to restart Splunk for the changes to take effect.

Additionally, you should check for mod time and privileges of that file. Some files do not get released as Splunk would expect, and therefore we will not recognize changes to the file. This situation has been known to occur when monitoring Windows IIS logs. In these scenarios, using the "alwaysOpenFile" parameter becomes necessary. Using this parameter is not recommended.