How to update lookup tables conditionally

[hmallett]

I have a large lookup table which is periodically generated from indexed data by a saved search.
The saved search takes a considerable amount of time. The saved search rewrites the lookup each time (I.e. it doesn’t append).

The indexed data from which the lookup is generated is not updated continuously.

Every time the saved search is used to build the lookup runs, it uses a large amount of Splunk resources.
If there has been no new indexed data, this means that we rebuild the lookup, using all those resources,
when we could have simply done nothing instead.

Is there an elegant way to modify the saved search so that if the latest indexed data is newer than a field in the lookup, we rebuild the lookup, but if not, the saved search ends without changing the lookup?

[VatsalJagani]

@hmallett,

There might two possibilities in your case.

• You always want the results of saved search to be appended to lookup. In this case you can use append=true with outputlookup command.
• In another case you sometimes require to update the lookup based on some field. By this what I mean is you might have some unique field in the lookup and if same value pops up from the search you want to update the row in the lookup instead of appending new row. In this case you can create kv-store lookup and make the unique value as a key in kvstore lookup. You can find more about kvstore lookup, just google it.

Hope this helps!!!

[mmccul]

The approach I'd at least consider would be to construct an alert that determines if the lookup table needs modifying. Run that alert periodically, keep that alert query simple. To make this alert work, you need some kind of time column or similar method to know "here's how recently updated this lookup table is". If you can use the lookup table to return a timestamp, you could even have that construct an earliest= field specification for your main search for data, so that the search starts at the time of the lookup table update timestamp.

Then, you'd need a custom alert action that executes a non-scheduled saved search to perform the lookup table update. Since a custom alert action could be a script, it could be used to trigger the saved search that does the update.