Getting Data In

I'm testing splunk, and when I edit my logfiles, splunk doesn't notice the changes. Why?

jrodman
Splunk Employee
Splunk Employee

I have a test logfile I fed into Splunk:

Apr 13 10:41:16 support05 kernel: [1815783.556088] usb 2-1: new full speed USB device using uhci_hcd and address 32 Apr 13 10:41:16 support05 kernel: [1815783.699049] usb 2-1: not running at top speed; connect to a high speed hub

and so on.

Splunk consumed the file just fine.

Then I opened the file and overwrote some text in the middle of the file. Splunk ignored my changes. Why didn't splunk re-index those lines?

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

jrodman
Splunk Employee
Splunk Employee

Er, if the last place it was in the file changed, it reindex the whole file too. If they both match and there's new data, it starts from that offset. 😉

0 Karma

jrodman
Splunk Employee
Splunk Employee

Fwiw, in 4.1 it's changes in modification time or file size.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...