Getting Data In

force Splunk to start at the end of a file instead of where it left off after shutdown

Path Finder

The ability for Splunk to start where it left off is a great feature. However, sometimes that feature hurts us.

Scenario: Indexer detects an error in an application that is located in Forwarder_123's logs. The Indexer kicks off a bash script that shuts down Splunk on Forwarder_123. (we shut it down because the application's error could exceed our license max in a matter of minutes) To fix the application, we restart the Application Server, so after the App Server restarts, it may be 30 minutes from the initial error was found. If we restart Splunk normally on Forwarder_123, all of the data that we do not want will be sent to the Indexer, which we do not want.

So, is there a way to tell the Splunk forwarder to start reading log entries from the last timestamp, instead of from the last location it remembers reading from?

I have tried the following command, but it does not appear to be what I want:

/opt/splunk/server/splunk/bin/splunk clean eventdata -f

The forwarder does NOT index any data. It simply sends the data straight to the Indexer.

Thanks,
Sean

Tags (1)
1 Solution

Splunk Employee
Splunk Employee

Hello Sean,

It sounds like you are looking for followTail in inputs.conf:

http://www.splunk.com/base/Documentation/latest/admin/Inputsconf

followTail = [0|1]
* Determines whether to start monitoring at the beginning of a file or at the end (and then index all events 
  that come in after that). 
* If set to 1, monitoring begins at the end of the file (like tail -f).
* If set to 0, Splunk will always start at the beginning of the file. 
* This only applies to files the first time Splunk sees them. After that, Splunk's internal file position 
  records keep track of the file. 
* Defaults to 0.

View solution in original post

Splunk Employee
Splunk Employee

Hello Sean,

It sounds like you are looking for followTail in inputs.conf:

http://www.splunk.com/base/Documentation/latest/admin/Inputsconf

followTail = [0|1]
* Determines whether to start monitoring at the beginning of a file or at the end (and then index all events 
  that come in after that). 
* If set to 1, monitoring begins at the end of the file (like tail -f).
* If set to 0, Splunk will always start at the beginning of the file. 
* This only applies to files the first time Splunk sees them. After that, Splunk's internal file position 
  records keep track of the file. 
* Defaults to 0.

View solution in original post

Splunk Employee
Splunk Employee

Right. It should act like tail -f, and only pick up the last lines of the file. However, moving forward after that point, it would always pick up in the same place it left off.

0 Karma

Path Finder

This is just a forwarder (no indexing occurs on it), so if I completely uninstalled Splunk, and then reinstalled Splunk with the followTail=1, the changes should be picked up, correct?

Thanks,
Sean

0 Karma

Splunk Employee
Splunk Employee

by the way, the clean command is for cleaning indexes, you could clean out ALL the data from an index by doing 'clean eventdata -f'

0 Karma

Splunk Employee
Splunk Employee

You could delete the file or change the first 256 bytes of the file, causing the crc to change and the file to be reindexed, starting at the end.

However, after that, even if you removed the file, the fishbucket would still keep track of the file and pick up where it left off.

0 Karma

Path Finder

So if it happens the first time Splunk sees the file, would I need to remove the entry from inputs.conf, restart Splunk, and then add the entries back into the inputs.conf along with 'followTail = 1'?

Does that sound right, or does "only applies to files the first time Splunk sees them" mean something else? (like maybe on restart)

Thanks,
Sean

0 Karma