Getting Data In

I'm testing splunk, and when I edit my logfiles, splunk doesn't notice the changes. Why?

jrodman
Splunk Employee
Splunk Employee

I have a test logfile I fed into Splunk:

Apr 13 10:41:16 support05 kernel: [1815783.556088] usb 2-1: new full speed USB device using uhci_hcd and address 32 Apr 13 10:41:16 support05 kernel: [1815783.699049] usb 2-1: not running at top speed; connect to a high speed hub

and so on.

Splunk consumed the file just fine.

Then I opened the file and overwrote some text in the middle of the file. Splunk ignored my changes. Why didn't splunk re-index those lines?

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

View solution in original post

jrodman
Splunk Employee
Splunk Employee

Er, if the last place it was in the file changed, it reindex the whole file too. If they both match and there's new data, it starts from that offset. 😉

0 Karma

jrodman
Splunk Employee
Splunk Employee

Fwiw, in 4.1 it's changes in modification time or file size.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!