- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Re: Indexing salt on ID value
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I'm looking for a way to not index an event if the ID is already in the index.
The log will have this format :
Unique ID;data;data2;etc..
Unique ID2;data3;data4;etc..
but two different log files may contain the same event. Is there a way to use the Unique ID as a salt ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.
The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.
If anyone else knows better ways of doing this, I would be very interested as well.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As the field extraction will take place after the event is indexed there is no way of checking if the ID has already been captured. At least not with standard Splunk methods. "Salting" in Splunk can only be done with complete files, not with parts of an event inside this file.
The only way I could think of would be an external script which processes the input files, runs a search against Splunk to see if the ID can be found, and only if not will forward the log line to another file, which then can be indexed by Splunk.
If anyone else knows better ways of doing this, I would be very interested as well.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's what I thought.
Thanks anyway for your answer.
Update : Another workaroud would be to generate one file by event with the ID in the filename in order to salt the filename.
But can Splunk delete files which it didn't index ? Since the 'correct' files are deleted after indexing, I don't want to have the 'incorrect' one to stack on the folder.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you are using a sinkhole policy, Splunk will delete the indexed files, but not the "incorrect" ones. These however could be moved with a cronjob...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the confirmation !
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you could mark the answer as "accepted" I would highly appreciate it. This will also show others that there's a valid solution.
Splunk Platform | Upgrading your Splunk Deployment to Python 3.9
From Product Design to User Insights: Boosting App Developer Identity on Splunkbase
Detect and Resolve Issues in a Kubernetes Environment
