topic Re: Conditional Outputlookup - dont write when no results in Splunk Search

Why do daily outputlookup searches have zero results and leave an empty lookup file?

pj — Tue, 24 May 2022 18:41:02 GMT

We often create daily lookups from our search results, which are then used for several other key searches. On occasion, upstream processes fail and data that we index into splunk comes in late for whatever reason or doesnt come in at all. The problem is that if our daily outputlookup searches run and pick up zero results they wipe out the lookup file completely and leave us with an empty lookup file.

Looking to do something along these lines:

if event count is less than xxx, then do nothing, 
else
write outputlookup

I am open to other suggestions also to get around this issue. I dont want to append and if the results do come in then I do need to lose the results in the file from the previous day.

Thanks!

Re: Conditional Outputlookup - dont write when no results

woodcock — Sat, 27 Jun 2015 03:45:53 GMT

This will do it, but it necessitates running your search twice:

Your Search Here That Creates "count" | addinfo | eval info_max_time=if((count<xxx), 0, info_max_time) | map search="search earliest=$info_min_time$ latest=$info_max_time$ Your Search Here That Creates "count" | outputlookup SomeFile.csv"

What happens is that passing in 0 for latest causes the search to crash before it gets to the outputlookup command when count < xxx.

Re: Conditional Outputlookup - dont write when no results

koshyk — Tue, 04 Sep 2018 13:31:08 GMT

From Splunk 7.1.x , there is a parameter called override_if_empty which does exactly the same (but only when it is empty)

For Previous version 6.x etc. what I have done is two level steps (You could do in one single step if your search is simple). Let's say your old csv is called my_original.csv
1. Write the outputlookup csv based on current data. Call this by another name my_temp.csv
2. Do an append condition and check for valid records on my_temp.csv. If not valid, use the my_original.csv.

Below is an example for checking if there are more than 5 records within the my_temp.csv, otherwise use existing my_original.csv

| inputlookup my_temp.csv
| eval temp="true"
| append [|inputlookup my_original.csv |  eval original="true"]
| eventstats count(temp) as tempCount
| where (tempCount>5 and temp="true") OR ((isnull(tempCount) OR (tempCount<=5)) AND original="true")
| outputlookup my_original.csv

Re: Conditional Outputlookup - dont write when no results

brannonrad — Thu, 15 Nov 2018 02:13:53 GMT

I had a similar problem - I have some key lookup files that I never want to be empty, because when they become empty, they can wreak havoc on my dashboards! Reasons why they might become empty: low disk space or application issues (i.e., not normal situations). To protect against this, I modified my automated lookup searches as follows:

<search that builds my_lookup.csv> | eval new=1 | append [|inputlookup my_lookup.csv | eval new=0] | eventstats sum(new) as newCount | eval useNum=if(newCount>5,1,0) | where new=useNum | fields - new newCount useNum | outputlookup my_lookup.csv

What I like about my solution is it will exclusively use the new data if it exists and fall back to the old data if the new data has less than 5 events (i.e., there is no "merging" of the new and old data, which can cause issues if data has been deleted.)

Re: Conditional Outputlookup - dont write when no results

woodcock — Sat, 13 Apr 2019 16:47:19 GMT

Hey, @pj, come back and Pick an answer to Accept to close your question. There are many good answers here, so you might consider an UpVote or 2, also.

Re: Conditional Outputlookup - dont write when no results

simonverzijl — Mon, 15 Jul 2019 07:43:12 GMT

Clever solution, took a minute before I got it 😉

Re: Conditional Outputlookup - dont write when no results

aa70627 — Fri, 26 Feb 2021 21:17:22 GMT

@woodcock that's a creative solution ! Nice

Re: Conditional Outputlookup - dont write when no results

twollenslegel_s — Tue, 24 May 2022 17:54:25 GMT

Always consider creating an ideas post if you see something you want.

https://ideas.splunk.com/ideas/EID-I-1474

Re: Conditional Outputlookup - dont write when no results

twollenslegel_s — Tue, 24 May 2022 18:10:07 GMT

Another option i was just made aware of that will work today:

This will just create the file if you have > 5000 but less than <10000 events, but can be adjusted to whatever values make sense.

<populating search>  | eventstats count as table_length
| where table_length > 5000 and table_length < 10000 | fields - table_length | outputlookup create_empty=false lookup.csv