Knowledge Management

How to update events with new key/value pairs on a per event basis?

splunker1981
Path Finder

Hi Splunkers,

I was wondering if someone could shed some insight on whether this is even possible with Splunk, if so, what is the right approach to accomplish the following.

I'm using dummy data as an example, shown below. We have data which we import and for the most part only has one or two fields populated at ingest time. What I'm trying to figure out is how to go about updating the key/value pairs for specific fields on a per event basis.

Using the example below the first column will always be populated when we ingest the data. What I'd like to be able to do is as we discover additional values I'd like to be able to update the specific event with the new key/value. Let's say for example I have an event Milk and the only thing I know at ingest is the price. This would be the last event. I'd like to be able to go in and have the ability to populate the event with the quantity, vendor and date. All of which in the example below are blank initially. Is this something that is supported by Splunk? Thanks for the help in advanced.

Type         Vendor         Count       Price       Date
Muffin       abc             7.0        4.99        2016-09-01          
tart         xyz            10.0        1.99        2016-09-01
eggs         xyy            12.0        0.99        2016-09-01
milk                                    1.99       
0 Karma
1 Solution

somesoni2
Revered Legend

An alternative is to setup KV Store (assuming you don't have ton of data like in millions) for this data and update the KV Store data as and when you receive new field values. They are basically lookups and support updates based on key-value pair. See these links for more information. Keep in mind, you'd need a scheduled search or something to scan new data and update new field values to KV store.

http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZK
https://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/ConfigureKVstorelookups
http://dev.splunk.com/view/webframework-tutorials/SP-CAAAEZT

View solution in original post

somesoni2
Revered Legend

An alternative is to setup KV Store (assuming you don't have ton of data like in millions) for this data and update the KV Store data as and when you receive new field values. They are basically lookups and support updates based on key-value pair. See these links for more information. Keep in mind, you'd need a scheduled search or something to scan new data and update new field values to KV store.

http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZK
https://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/ConfigureKVstorelookups
http://dev.splunk.com/view/webframework-tutorials/SP-CAAAEZT

splunker1981
Path Finder

Thanks, I played around with this and seems to do the trick. How it scales is another question...

0 Karma

sundareshr
Legend

Splunk does not support modifying indexed data. What you could do, is ingest another event of Type=Milk, Vendor=MilkVendor, Date=2016-09-02 and generate a report that merges Price=1.99 for display purposes. In other words, let's assume you have the following two events ingested.

Type=Milk, Price=1.99
Type=Milk, Vendor=MilkVendor, Date=2016-09-02

A search like this, would give you your desired output

base search | stats last(Vendor) as Vendor last(Price) as Price last(Date) as Date by Type
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...