Getting Data In

Unique Contraint in Splunk

akshgpt25
Explorer

We have a number of files containing events. Each event has a unique id in itself. but the same event(with the same event_id) can exist in other files as well. We want 1 event to be indexed 1 time in Splunk and if the event with the same event_id comes again just avoid it. This is the same as _id in Elastic Search or another database's primary key.

Is it possible to achieve the same in Splunk? If yes, How?

 

 

Labels (1)
Tags (1)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

I'm not sure if this can first validate if an event is in Splunk before it routes data to Splunk and I don't know a great deal about it, but the DSP component may do what you want

https://www.splunk.com/en_us/software/stream-processing.html

 

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@akshgpt25 

Not sure it is possible during indexing, but you can get unique event in SPL by using dedup command. 

https://docs.splunk.com/Documentation/Splunk/8.0.4/SearchReference/Dedup

Thanks

0 Karma

akshgpt25
Explorer

Thanks, @kamlesh_vaghela 

But we have millions of events with this behavior and every time end-user have to use dedup in all their queries.

Is there any other way to achieve the same?

0 Karma

gcusello
Legend

Hi @akshgpt25 ,

as @kamlesh_vaghela  said it isn't possible to filter events before indexing to avoid duplicates, but you could ingest all the data and schedule a search that extract the unduplicated events and store them in a summary_index, then you can use this index for your searches.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...