topic Re: Clean up ip_intel kvstore in Splunk Search

Clean up ip_intel kvstore

weetabixsplunk — Mon, 19 Oct 2020 14:56:00 GMT

Hi guys,

I can see how this question comes across as dumb but I would like to remove duplicated entries from my ip_intel kv store.

I understand the whole purpose of a kv_store is used for data that constantly gets updated. I am finding lots of duplicated IPs in my ip_intel kvstore and I'd like to know if there's a better way to do it than just

| inputlookup ip_intel | dedup ip

I'd like to cleanup the file from duplicates and not just create a dedup search.

Thanks! 🙂

Re: Clean up ip_intel kvstore

inventsekar — Mon, 19 Oct 2020 15:39:46 GMT

Hi @weetabixsplunk i am not sure of this ip_intel kvstore, but, in first place how the duplicate entries entered this kvstore and can not you simply search the duplicate entries and delete them, i am not sure, just asking your ideas, suggestions.

Re: Clean up ip_intel kvstore

Azeemering — Mon, 19 Oct 2020 15:44:47 GMT

Normally what I would do to keep the intel kvstores clean and fresh is to limit the age of the entries by running a daily cleanup job:

A simple vanilla flavored example:

|inputlookup ip_intel |eval treshold = now()-86400 |where time>treshold ||outputlookup ip_intel