Splunk Search

Why do daily outputlookup searches have zero results and leave an empty lookup file?

pj
Contributor

We often create daily lookups from our search results, which are then used for several other key searches. On occasion, upstream processes fail and data that we index into splunk comes in late for whatever reason or doesnt come in at all. The problem is that if our daily outputlookup searches run and pick up zero results they wipe out the lookup file completely and leave us with an empty lookup file.

Looking to do something along these lines:

if event count is less than xxx, then do nothing, 
else
write outputlookup

I am open to other suggestions also to get around this issue. I dont want to append and if the results do come in then I do need to lose the results in the file from the previous day.

Thanks!

Labels (1)

twollenslegel_s
Splunk Employee
Splunk Employee

Another option i was just made aware of that will work today:

This will just create the file if you have > 5000 but less than <10000 events, but can be adjusted to whatever values make sense. 

<populating search>  | eventstats count as table_length
| where table_length > 5000 and table_length < 10000 | fields - table_length | outputlookup create_empty=false lookup.csv

 

0 Karma

twollenslegel_s
Splunk Employee
Splunk Employee

Always consider creating an ideas post if you see something you want. 

https://ideas.splunk.com/ideas/EID-I-1474

 

0 Karma

woodcock
Esteemed Legend

Hey, @pj, come back and Pick an answer to Accept to close your question. There are many good answers here, so you might consider an UpVote or 2, also.

0 Karma

brannonrad
Explorer

I had a similar problem - I have some key lookup files that I never want to be empty, because when they become empty, they can wreak havoc on my dashboards! Reasons why they might become empty: low disk space or application issues (i.e., not normal situations). To protect against this, I modified my automated lookup searches as follows:

<search that builds my_lookup.csv> | eval new=1 | append [|inputlookup my_lookup.csv | eval new=0] | eventstats sum(new) as newCount | eval useNum=if(newCount>5,1,0) | where new=useNum | fields - new newCount useNum | outputlookup my_lookup.csv

What I like about my solution is it will exclusively use the new data if it exists and fall back to the old data if the new data has less than 5 events (i.e., there is no "merging" of the new and old data, which can cause issues if data has been deleted.)

simonverzijl
Engager

Clever solution, took a minute before I got it 😉

0 Karma

koshyk
Super Champion

From Splunk 7.1.x , there is a parameter called override_if_empty which does exactly the same (but only when it is empty)

For Previous version 6.x etc. what I have done is two level steps (You could do in one single step if your search is simple). Let's say your old csv is called my_original.csv
1. Write the outputlookup csv based on current data. Call this by another name my_temp.csv
2. Do an append condition and check for valid records on my_temp.csv. If not valid, use the my_original.csv.

Below is an example for checking if there are more than 5 records within the my_temp.csv, otherwise use existing my_original.csv

| inputlookup my_temp.csv
| eval temp="true"
| append [|inputlookup my_original.csv |  eval original="true"]
| eventstats count(temp) as tempCount
| where (tempCount>5 and temp="true") OR ((isnull(tempCount) OR (tempCount<=5)) AND original="true")
| outputlookup my_original.csv

woodcock
Esteemed Legend

This will do it, but it necessitates running your search twice:

Your Search Here That Creates "count" | addinfo | eval info_max_time=if((count<xxx), 0, info_max_time) | map search="search earliest=$info_min_time$ latest=$info_max_time$ Your Search Here That Creates "count" | outputlookup SomeFile.csv"

What happens is that passing in 0 for latest causes the search to crash before it gets to the outputlookup command when count < xxx.

aa70627
Communicator

@woodcock that's a creative solution ! Nice

0 Karma
Get Updates on the Splunk Community!

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...

Want to Reduce Costs, Mitigate Risk, Improve Performance, or Increase Efficiencies? ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...