Getting Data In

Why won't Splunk re-index my data?

Mick
Splunk Employee
Splunk Employee

I wanted to see how Splunk would index my data, so I configured it to index a few files into a 'test' index. Now that I have it configured properly, I want to re-index that same data into the 'main' index. I cleaned the test index - ./splunk clean eventdata index test - and removed the index = test from inputs.conf, but Splunk doesn't automatically re-index the files - why not?

Tags (3)
1 Solution

amrit
Splunk Employee
Splunk Employee

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

View solution in original post

amrit
Splunk Employee
Splunk Employee

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

gkanapathy
Splunk Employee
Splunk Employee

i forget one very. very.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

that would be a very very very good feature to have.

0 Karma

amrit
Splunk Employee
Splunk Employee

maybe you should RTFM and get it right the first time. jk!! ❤️ ...did you clean before you had index=test in the conf? if you cleaned test, restarted, added index=test, and restarted again, you'll encounter the above fail. anyway, good point about finalizing in a different index. right now this isn't possible (although, you could add a bogus crcSalt and leave it there forever...), but we can add something. the idea would be something like "splunk reset filepos /path/to/file", which would use btprobe to zero out our record of the file. this feature does not exist currently.

Mick
Splunk Employee
Splunk Employee

even if I leave 'index = test' the files don't get re-read, but based on what you're saying, I would have to create a brand new index for the data to get re-read? That doesn't make sense to me, I want to add it to my existing index once I'm happy that it will be indexed correctly. What if I have to tweak settings several times before I get it right, do I need to create a new 'test' index each time?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...