I'm testing splunk, and when I edit my logfiles, splunk doesn't notice the changes. Why?

[jrodman]

I have a test logfile I fed into Splunk:

Apr 13 10:41:16 support05 kernel: [1815783.556088] usb 2-1: new full speed USB device using uhci_hcd and address 32 Apr 13 10:41:16 support05 kernel: [1815783.699049] usb 2-1: not running at top speed; connect to a high speed hub

and so on.

Splunk consumed the file just fine.

Then I opened the file and overwrote some text in the middle of the file. Splunk ignored my changes. Why didn't splunk re-index those lines?

[gkanapathy · Accepted solution]

By default, Splunk detects changes in files by first checking the modification time.

• If that has changed, it checks the first 256 bytes of the file
  • If that is different from the last time it saw the file, it will index the file from the beginning.
  • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
    • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

[jrodman]

Er, if the last place it was in the file changed, it reindex the whole file too. If they both match and there's new data, it starts from that offset. 😉

[jrodman]

Fwiw, in 4.1 it's changes in modification time or file size.