Knowledge Management

Am I able to use my own custom key for KV Store and update only partial values?

goodsellt
Contributor

I'm looking to use the KV store to power a table to track usage and ownership of IP addresses across our environment, as well as a time based metric when when we send userid and ipaddress mappings to our Palo Alto firewalls.

The sticking point I'm having is with the key, as I'd like the list of IP addresses to be the 'primary key' as it were, with one scheduled search I'm running updating the ownership records, while another external script would be syslogging the entries over to the firewalls and inserting an updated timestamp into the KV store.

I'd like to confirm I could do this. My two questions on the KV store are:
1. Am I able to use my own data as the "_key" instead of Splunk's autogenerated keys?
2. Am I able to update records in the KV store partially? As in only updating one of the supporting fields for a record?

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hi goodsellt!

Yes, you can explicitly set the _key. Splunk only auto populates it if no _key is provided in the outputlookup

http://dev.splunk.com/view/SP-CAAAEZH

The key ID field
By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. When you use outputlookup to write to the KV Store, a key ID is autogenerated if you don't specify one explicitly. If you want to modify a specific record, you need to provide its key ID.

To view the key ID field in lookups, add "_key" to the fields_list in the lookup definition.

As for updating records, KVSTORE supports CRUD and allows you to update records. As for updating a single field in the record, you can essentially fudge that by rewriting the record entirely simply updating the fields you require.

http://dev.splunk.com/view/SP-CAAAEZH#examples

- MattyMo

View solution in original post

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi goodsellt!

Yes, you can explicitly set the _key. Splunk only auto populates it if no _key is provided in the outputlookup

http://dev.splunk.com/view/SP-CAAAEZH

The key ID field
By default, each KV Store record has a unique key ID, which is stored in the internal "_key" field. When you use outputlookup to write to the KV Store, a key ID is autogenerated if you don't specify one explicitly. If you want to modify a specific record, you need to provide its key ID.

To view the key ID field in lookups, add "_key" to the fields_list in the lookup definition.

As for updating records, KVSTORE supports CRUD and allows you to update records. As for updating a single field in the record, you can essentially fudge that by rewriting the record entirely simply updating the fields you require.

http://dev.splunk.com/view/SP-CAAAEZH#examples

- MattyMo
0 Karma

goodsellt
Contributor

Thanks! I have my custom key working successfully!

0 Karma

utkarsh21
New Member

How did you use a manual key? I am trying to do the same. Thanks in advance!

0 Karma

goodsellt
Contributor

When you are loading the data into the KVStore, you want to rename the field you're using as _key, therefore it will be used as the primary key record instead of what Splunk autogenerated for you.

I'm using a Python script to generate the documents I put into KVStore as part of another process, but an example of how I build the document is:
doc = dict(_key=str(item['IP']), User=str(item['User']), Domain=str(item['Domain']),
Last_Update=int(float(item['Last_Update'])), Last_PA_Send=int(0))

consider the item[xxxx] as my fields from my search results, since I wanted my IP field to be the primary key I renamed it to _key prior to sending it to KVStore.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...