Why won't Splunk re-index my data?

[Mick]

I wanted to see how Splunk would index my data, so I configured it to index a few files into a 'test' index. Now that I have it configured properly, I want to re-index that same data into the 'main' index. I cleaned the test index - ./splunk clean eventdata index test - and removed the index = test from inputs.conf, but Splunk doesn't automatically re-index the files - why not?

[amrit · Accepted solution]

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

[amrit]

See --reset @ http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/CommandlinetoolsforusewithSupport...

[gkanapathy]

i forget one very. very.

[gkanapathy]

that would be a very very very good feature to have.

[amrit]

maybe you should RTFM and get it right the first time. jk!! ❤️ ...did you clean before you had index=test in the conf? if you cleaned test, restarted, added index=test, and restarted again, you'll encounter the above fail. anyway, good point about finalizing in a different index. right now this isn't possible (although, you could add a bogus crcSalt and leave it there forever...), but we can add something. the idea would be something like "splunk reset filepos /path/to/file", which would use btprobe to zero out our record of the file. this feature does not exist currently.

[Mick]

even if I leave 'index = test' the files don't get re-read, but based on what you're saying, I would have to create a brand new index for the data to get re-read? That doesn't make sense to me, I want to add it to my existing index once I'm happy that it will be indexed correctly. What if I have to tweak settings several times before I get it right, do I need to create a new 'test' index each time?