Getting Data In

I'm testing splunk, and when I edit my logfiles, splunk doesn't notice the changes. Why?

jrodman
Splunk Employee
Splunk Employee

I have a test logfile I fed into Splunk:

Apr 13 10:41:16 support05 kernel: [1815783.556088] usb 2-1: new full speed USB device using uhci_hcd and address 32 Apr 13 10:41:16 support05 kernel: [1815783.699049] usb 2-1: not running at top speed; connect to a high speed hub

and so on.

Splunk consumed the file just fine.

Then I opened the file and overwrote some text in the middle of the file. Splunk ignored my changes. Why didn't splunk re-index those lines?

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

By default, Splunk detects changes in files by first checking the modification time.

  • If that has changed, it checks the first 256 bytes of the file
    • If that is different from the last time it saw the file, it will index the file from the beginning.
    • If it is the same as the last time it saw the file, it will check the last 256 bytes of the file.
      • If it has changed, Splunk will index new events from the point that previously read.

Thus, if you change the file in the middle, Splunk may detect the modification times, but will not see any change at the beginning or end of the file, and therefore will index any part of the file anew.

jrodman
Splunk Employee
Splunk Employee

Er, if the last place it was in the file changed, it reindex the whole file too. If they both match and there's new data, it starts from that offset. 😉

0 Karma

jrodman
Splunk Employee
Splunk Employee

Fwiw, in 4.1 it's changes in modification time or file size.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...