We often create daily lookups from our search results, which are then used for several other key searches. On occasion, upstream processes fail and data that we index into splunk comes in late for whatever reason or doesnt come in at all. The problem is that if our daily outputlookup searches run and pick up zero results they wipe out the lookup file completely and leave us with an empty lookup file.
Looking to do something along these lines:
if event count is less than xxx, then do nothing, else write outputlookup
I am open to other suggestions also to get around this issue. I dont want to append and if the results do come in then I do need to lose the results in the file from the previous day.
Another option i was just made aware of that will work today:
This will just create the file if you have > 5000 but less than <10000 events, but can be adjusted to whatever values make sense.
<populating search> | eventstats count as table_length | where table_length > 5000 and table_length < 10000 | fields - table_length | outputlookup create_empty=false lookup.csv
I had a similar problem - I have some key lookup files that I never want to be empty, because when they become empty, they can wreak havoc on my dashboards! Reasons why they might become empty: low disk space or application issues (i.e., not normal situations). To protect against this, I modified my automated lookup searches as follows:
<search that builds my_lookup.csv> | eval new=1 | append [|inputlookup my_lookup.csv | eval new=0] | eventstats sum(new) as newCount | eval useNum=if(newCount>5,1,0) | where new=useNum | fields - new newCount useNum | outputlookup my_lookup.csv
What I like about my solution is it will exclusively use the new data if it exists and fall back to the old data if the new data has less than 5 events (i.e., there is no "merging" of the new and old data, which can cause issues if data has been deleted.)
From Splunk 7.1.x , there is a parameter called
override_if_empty which does exactly the same (but only when it is empty)
For Previous version 6.x etc. what I have done is two level steps (You could do in one single step if your search is simple). Let's say your old csv is called
1. Write the outputlookup csv based on current data. Call this by another name
2. Do an append condition and check for valid records on
my_temp.csv. If not valid, use the
Below is an example for checking if there are more than 5 records within the
my_temp.csv, otherwise use existing
| inputlookup my_temp.csv | eval temp="true" | append [|inputlookup my_original.csv | eval original="true"] | eventstats count(temp) as tempCount | where (tempCount>5 and temp="true") OR ((isnull(tempCount) OR (tempCount<=5)) AND original="true") | outputlookup my_original.csv
This will do it, but it necessitates running your search twice:
Your Search Here That Creates "count" | addinfo | eval info_max_time=if((count<xxx), 0, info_max_time) | map search="search earliest=$info_min_time$ latest=$info_max_time$ Your Search Here That Creates "count" | outputlookup SomeFile.csv"
What happens is that passing in
latest causes the search to crash before it gets to the
outputlookup command when
count < xxx.