Knowledge Management

How should I implement a last_modified_timestamp for each row in a KV store?

kwana37
New Member

We correlate multiple data sources into a master lookup that also periodically gets exported and would like to implement a last_modified_timestamp field so only new rows/rows whose values have changed since the last export are exported. What is the best way to compare what is currently in the KV store to what is being inserted into it? We have a lot of data and a lot of fields, so efficiency is key.

0 Karma

jonmargulies
Path Finder

Is the KV updated by just a few well-maintained searches? i.e., will it only be updated in controllable, predictable ways?

If so, you can do this by adding a last_modified field to the KV (| inputlookup example_kv | eval last_modified=now() | outputlookup example_kv), and then keep it updated by making sure that all of your updating queries modify that last_modified field as well (using now() as the value).

If you do this, your updating query will go from something like this:
| inputlookup example_kv | search _key=544948df3ec32d7a4c1d9755 | eval CustName="Marge Simpson" | eval CustCity="Springfield" | outputlookup example_kv append=True

to something like this:
| inputlookup example_kv | search _key=544948df3ec32d7a4c1d9755 | eval CustName="Marge Simpson" | eval CustCity="Springfield" | eval last_modified=now() | outputlookup example_kv append=True

Then when you're exporting you can just search for ... | where last_modified>relative_time(now(), -1d@d) (adjust as needed).

0 Karma

starcher
SplunkTrust
SplunkTrust

Key is key. Meaning make sure you form your _key value for your rows such that when you update it updates the precise row then when you update a time based column with the latest time you should get the result you want.

0 Karma

kwana37
New Member

Hi starcher. Thanks for replying. We have already tried that, but because we are pulling data every day that may or may not be different, our "updates" are not true updates, so that solution was not sufficient for our purposes. We want to figure out whether the data for a row has truly changed.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...