Splunk Search

Indexing salt on ID value

arthurf
Explorer

Hello,

I'm looking for a way to not index an event if the ID is already in the index.

The log will have this format :
Unique ID;data;data2;etc..
Unique ID2;data3;data4;etc..

but two different log files may contain the same event. Is there a way to use the Unique ID as a salt ?

Tags (1)
0 Karma
1 Solution

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

View solution in original post

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

View solution in original post

arthurf
Explorer

That's what I thought.
Thanks anyway for your answer.

Update : Another workaroud would be to generate one file by event with the ID in the filename in order to salt the filename.
But can Splunk delete files which it didn't index ? Since the 'correct' files are deleted after indexing, I don't want to have the 'incorrect' one to stack on the folder.

0 Karma

DMohn
Motivator

If you are using a sinkhole policy, Splunk will delete the indexed files, but not the "incorrect" ones. These however could be moved with a cronjob...

0 Karma

arthurf
Explorer

Thanks for the confirmation !

0 Karma

DMohn
Motivator

If you could mark the answer as "accepted" I would highly appreciate it. This will also show others that there's a valid solution.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.