Splunk Search

Indexing salt on ID value

arthurf
Explorer

Hello,

I'm looking for a way to not index an event if the ID is already in the index.

The log will have this format :
Unique ID;data;data2;etc..
Unique ID2;data3;data4;etc..

but two different log files may contain the same event. Is there a way to use the Unique ID as a salt ?

Tags (1)
0 Karma
1 Solution

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

View solution in original post

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

arthurf
Explorer

That's what I thought.
Thanks anyway for your answer.

Update : Another workaroud would be to generate one file by event with the ID in the filename in order to salt the filename.
But can Splunk delete files which it didn't index ? Since the 'correct' files are deleted after indexing, I don't want to have the 'incorrect' one to stack on the folder.

0 Karma

DMohn
Motivator

If you are using a sinkhole policy, Splunk will delete the indexed files, but not the "incorrect" ones. These however could be moved with a cronjob...

0 Karma

arthurf
Explorer

Thanks for the confirmation !

0 Karma

DMohn
Motivator

If you could mark the answer as "accepted" I would highly appreciate it. This will also show others that there's a valid solution.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...