I'm running many scheduled searches. In SQL, there is the "on error" capability that let's you avoid taking further action if something broke before.
Many of our searches do something like the following:
|dbxquery connection=xxx query="select * from tablex"
|massage the data
|outputlookup tablex_fast_lookup
If the SQL command dies due to a SQL issue, then we end up writing an empty lookup file. Once the lookup file is empty, dozens of other things break. I suppose I could convert it to a kv-store and merge data to the lookup with a last update timestamp. Then we could purge older data based on the timestamp.
I figured that if I could abort the command, then the lookup file would have stale data, but it would not be empty.
... View more