Splunk Search

Can I update KV store through a search?

pjvarjani
Path Finder

Hi,

I am doing a POC to check if KV Store is a better option as compared to a traditional file lookup for my app. So I did the following things.

  1. Initialized KV Store through collection.conf
  2. removed all existing file-based lookups.
  3. changed searches in savedsearch.conf to populate KV Store instead of a traditional lookup.

I am able to configure & use KV Store without any issue. But how I am getting a benefit here from using KV Store?

Let's say one of my saved search is scheduled to execute at every hour and dump hundreds of result to lookup/KV Store. Now I know each time my search executes 90% of data is going to be the similar what I have dumped previously in my lookup/KV Store. In traditional lookup I don't have option to update my lookup with selective(modified) fields. I know KV Store is gives flexibility of updating only certain records stored in it. But is it possible to update only few records in KV Store using search query?

Thanks,
Pankaj

0 Karma

nekbote
Path Finder

If Key field of the KVstore is unique, even though 90% of data is same, hourly search which feds data to KVstore will only add the10% that isn't already in KVStore and would update 90% which are existing if there has been change in any of their field values. Below is sample command to update KVStore using the hourly search

index=xyz |eval key=employeeid."-".employeedept |table employeeid,employeedept,key|outputlookup EmployeKVStore key_field=key append=true

mrgibbon
Contributor

In theory you should be able to take your new data, search the KVStore for anything that doesn't match and then use the outputlookup command to store the new data inside the KVStore.
I cant help right now with the specific commands, but that approach should work.

http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZH

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...