Hi - I am try to build a saved search that has conditions so that the full search only executes when all conditions are met.
I have several file feeds that are sent to Splunk weekly. However their timing is sporadic so I can't rely on a set time.
Once a new data feed arrives we run a search which joins data from other indexes and adds enrichment via lookups to create a transformed data set. This transformed data set is then sent to a separate index which has an accelerated data model in place.
My idea is that I schedule the search to run daily and only runs the full search when a new file has arrived.
So the first condition is to check the latest source file in the landing index against the latest source file in the accelerated index.
If the two are different then I want to proceed with the search logic to transform the data set.
Note in the example below the source file contains DDMM to signify the date of the feed so I use a wildcard here
| tstats latest where index=orig_index source=sourcefile*.txt earliest=-30d@d latest=now by time , source
| sort 1 - _time
| fields - _time
| appendcols
[| tstats latest(baseset.source) as orig_source from datamodel=datamodelname where baseset.source=sourcefile*.txt earliest=-30d@d latest=now ]
This give me the latest source from the landing index and the latest source in the accelerated index
I now add a where clause to act as the first condition. I've tested this and it works in that I can add search statements after the where clause and they are not executed if the two sources match.
where source!=orig_source
However I actually want to now execute the main search logic and return the data set in the search. The only way I can do this as far as I can tell is to use append. The issue I am seeing however is that append runs regardless of whether the where condition is met or not.
I suspect this is because append runs as a sub-search. Can anyone think of a way around this?
Here's the full search
| tstats latest where index=orig_index source=sourcefile*.txt earliest=-30d@d latest=now by time , source
| sort 1 - _time
| fields - _time
| appendcols
[| tstats latest(baseset.source) as orig_source from datamodel=datamodelname where baseset.source=sourcefile*.txt earliest=-30d@d latest=now ]
| where source!=orig_source
| append [|savedsearch "dcv_accelerate_search"]
