Splunk Search

How to update lookup tables conditionally

hmallett
Path Finder

I have a large lookup table which is periodically generated from indexed data by a saved search.
The saved search takes a considerable amount of time. The saved search rewrites the lookup each time (I.e. it doesn’t append).

The indexed data from which the lookup is generated is not updated continuously.

Every time the saved search is used to build the lookup runs, it uses a large amount of Splunk resources.
If there has been no new indexed data, this means that we rebuild the lookup, using all those resources,
when we could have simply done nothing instead.

Is there an elegant way to modify the saved search so that if the latest indexed data is newer than a field in the lookup, we rebuild the lookup, but if not, the saved search ends without changing the lookup?

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@hmallett,

There might two possibilities in your case.

  • You always want the results of saved search to be appended to lookup. In this case you can use append=true with outputlookup command.
  • In another case you sometimes require to update the lookup based on some field. By this what I mean is you might have some unique field in the lookup and if same value pops up from the search you want to update the row in the lookup instead of appending new row. In this case you can create kv-store lookup and make the unique value as a key in kvstore lookup. You can find more about kvstore lookup, just google it.

Hope this helps!!!

0 Karma

mmccul
SplunkTrust
SplunkTrust

The approach I'd at least consider would be to construct an alert that determines if the lookup table needs modifying. Run that alert periodically, keep that alert query simple. To make this alert work, you need some kind of time column or similar method to know "here's how recently updated this lookup table is". If you can use the lookup table to return a timestamp, you could even have that construct an earliest= field specification for your main search for data, so that the search starts at the time of the lookup table update timestamp.

Then, you'd need a custom alert action that executes a non-scheduled saved search to perform the lookup table update. Since a custom alert action could be a script, it could be used to trigger the saved search that does the update.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...