Indexing salt on ID value

[arthurf]

Hello,

I'm looking for a way to not index an event if the ID is already in the index.

The log will have this format :
Unique ID;data;data2;etc..
Unique ID2;data3;data4;etc..

but two different log files may contain the same event. Is there a way to use the Unique ID as a salt ?

[DMohn · Accepted solution]

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

[arthurf]

That's what I thought.
Thanks anyway for your answer.

Update : Another workaroud would be to generate one file by event with the ID in the filename in order to salt the filename.
But can Splunk delete files which it didn't index ? Since the 'correct' files are deleted after indexing, I don't want to have the 'incorrect' one to stack on the folder.

[DMohn]

If you are using a sinkhole policy, Splunk will delete the indexed files, but not the "incorrect" ones. These however could be moved with a cronjob...

[arthurf]

Thanks for the confirmation !

[DMohn]

If you could mark the answer as "accepted" I would highly appreciate it. This will also show others that there's a valid solution.