Getting Data In

Splunk stopped indexing file

kaufmanm
Communicator

After upgrading a Solaris SPARC forwarder from Splunk 3.4.9 to 4.1.4 (build 82143) one log file stopped being indexed. Lots of new data is being written to it, but I'm not seeing it on the indexer. Ten plus other files are being monitored fine from the same forwarder. In inputs.conf it's:

[monitor:///log/syslog/network/netscalar.log]
disabled = false

In the TailingProcessor:FileStatus it's:

log/syslog/network/netscalar.log    
file position   10184387
file size   10184387
percent 100.00
type    open file

The log file is rotated, so there is also netscalar.log.1 and so on in the directory. I tried clearing the eventdata in the fishbucket, which took a few hours but it read all the files again, but that didn't fix it. Is there a way I can get Splunk to treat this file as entirely new to get this data indexed?

Tags (1)
0 Karma
1 Solution

briang67
Communicator

I've seen issues in the past where a file stops forwarding because of the rotation strategy - whether the file is moved (inode change) or copied and truncated in place. Not sure if that's still an issue with the newest versions of splunk, but we used to handle this situation by splunking the directory and then whitelisting for the specific file.

View solution in original post

kaufmanm
Communicator

I deleted the log file and created a new one and Splunk is indexing it fine, thanks all set.

0 Karma

briang67
Communicator

I've seen issues in the past where a file stops forwarding because of the rotation strategy - whether the file is moved (inode change) or copied and truncated in place. Not sure if that's still an issue with the newest versions of splunk, but we used to handle this situation by splunking the directory and then whitelisting for the specific file.

kaufmanm
Communicator

I changed inputs.conf to monitor the directory and _whitelist the file, but it still shows up as 100% read in the TailingProcessor:FileStatus. FWIW, the file size there is the same as the actual file size on the disk.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...