Getting Data In

Rolled log file not being indexed

Jaci
Splunk Employee
Splunk Employee

I monitor a log file (access_log) that gets rolled every night at 1 am using a copy command "cp /dev/null access_today". After the file rolls splunk randomly stops indexing the file until a day or two later after a roll. This only happens for this log file.

The log file is empty after the copy but usually has data within a minute or so. Is it possible that the Splunk database has an zero size CRC entry of this file? How can we check that?

I'm running splunk 4.0.9.

I have this set in my inputs.conf

[monitor:///host1/log/]  
crcSalt=<SOURCE>    
_whitelist = (access_log|error_log)

Here is a sample of the access_log file:

192.7.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 382777 - - - - 192.33.220.7 -
192.5.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 309851 - - - - 192.33.220.7 -
192.6.12.120 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 319522 - - - - 192.33.220.7 -
- - - [11/May/2010:01:35:34 -0400] 200 349 - - GET /xxx/xxx.html HTTP/0.9 754 - - - - 192.33.220.7 -
192.7.12.10 - - [11/May/2010:01:35:34 -0400] 200 - - XmlPipe POST /xxx/xxx/xxx HTTP/1.0 396867 - - - - 192.33.220.7 -
Tags (2)
1 Solution

Simeon
Splunk Employee
Splunk Employee

If you turn on the debug flags for file monitoring, Splunk will tell you the status of that file. You can review how to set these flags here:

http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs

Specifically, you should edit the $SPLUNK_HOME/etc/log.cfg file and set the following parameters:

category.FileInputTracker=DEBUG
category.selectProcessor=DEBUG
category.TailingProcessor=DEBUG

You will need to restart Splunk for the changes to take effect.

Additionally, you should check for mod time and privileges of that file. Some files do not get released as Splunk would expect, and therefore we will not recognize changes to the file. This situation has been known to occur when monitoring Windows IIS logs. In these scenarios, using the "alwaysOpenFile" parameter becomes necessary. Using this parameter is not recommended.

View solution in original post

Lowell
Super Champion

I assume that your cp command it to essentially to truncate your log file because you don't want to restart your server? Any chance that you could use logrotate (with copytruncate mode) or use a pipe-based file rotator like rotatelogs (comes with apache I believe) instead?

0 Karma

Lowell
Super Champion

Out of curiosity, why do you have the crcSalt=<SOURCE> in there?

0 Karma

Simeon
Splunk Employee
Splunk Employee

If you turn on the debug flags for file monitoring, Splunk will tell you the status of that file. You can review how to set these flags here:

http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs

Specifically, you should edit the $SPLUNK_HOME/etc/log.cfg file and set the following parameters:

category.FileInputTracker=DEBUG
category.selectProcessor=DEBUG
category.TailingProcessor=DEBUG

You will need to restart Splunk for the changes to take effect.

Additionally, you should check for mod time and privileges of that file. Some files do not get released as Splunk would expect, and therefore we will not recognize changes to the file. This situation has been known to occur when monitoring Windows IIS logs. In these scenarios, using the "alwaysOpenFile" parameter becomes necessary. Using this parameter is not recommended.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...