Splunk Search

Indexing salt on ID value

arthurf
Explorer

Hello,

I'm looking for a way to not index an event if the ID is already in the index.

The log will have this format :
Unique ID;data;data2;etc..
Unique ID2;data3;data4;etc..

but two different log files may contain the same event. Is there a way to use the Unique ID as a salt ?

Tags (1)
0 Karma
1 Solution

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

View solution in original post

DMohn
Motivator

As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.

The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.

If anyone else knows better ways of doing this, I would be very interested as well.

arthurf
Explorer

That's what I thought.
Thanks anyway for your answer.

Update : Another workaroud would be to generate one file by event with the ID in the filename in order to salt the filename.
But can Splunk delete files which it didn't index ? Since the 'correct' files are deleted after indexing, I don't want to have the 'incorrect' one to stack on the folder.

0 Karma

DMohn
Motivator

If you are using a sinkhole policy, Splunk will delete the indexed files, but not the "incorrect" ones. These however could be moved with a cronjob...

0 Karma

arthurf
Explorer

Thanks for the confirmation !

0 Karma

DMohn
Motivator

If you could mark the answer as "accepted" I would highly appreciate it. This will also show others that there's a valid solution.

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...