Splunk Search

Sub Search failing due to return with no items

jmo1
Path Finder

I am very close but need some assistance.  I am attempting to create an alert based upon the criteria "Free Megabytes"<6000 AND "% Free Space" <10.  I have that logic working below...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 | return 1000 host ]

 

The above code works as long as there is a hit for the Free Megabytes < 6000.  However if there are no hits, no host is returned to the % Free Space so it show all hosts that meet that critera.  How can this be adapted so that no hosts returned doesn't result in further query?  I am guessing eval, but my Splunk-fu is weak.

 

Any help is appreciated but actual code would be most helpful.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try this on pre-8.1 Splunk

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ] ]

and try this on Splunk 8.1

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]
---
If this reply helps you, Karma would be appreciated.
0 Karma

jmo1
Path Finder

Thank you so much.  We are on 8.1, so I read up on the REQUIRE command.

I am seeing something very strange so maybe there is something else going on.

search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ]

 

When I run the search above and when I run the 8.1 version with Require I get no returned items.  As I would expect.  However, if I change Value<6000 to Value<2000, I get incorrect results.  I don't understand how this can be because if it is less than 6000, it is also less than 2000.  Any ideas of what could be causing this? 

0 Karma

jmo1
Path Finder

Let me be more clear...

When I run...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]

I get ZERO events, which is expected.

However, if I change the 6000 to a 2000, I get events returned.

 

To further diagnose, here is what I am seeing...

If I run the subsearch...

host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require

I get servers returned.

If I change to 2000, no servers returned.

Therefore, it seems something in the first part is not handling correctly when no servers are returned.  

NOTE: I used the other (pre-8.1 logic) and it exhibits the exact same returns.

 

It appears with no returns from the subsearch, the first search is running against all servers.  I need the first search to fail if the subsearch returns nothing (which it seems is what the require command is suppost to do).

 

Thanks.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

"if it is less than 6000, it is also less than 2000" ? 5999 is less than 6000 but it isn't less than 2000!

0 Karma

jmo1
Path Finder

Did you even look at the code or what I am saying?  The query for less than 6000 returned nothing, but the query for less than 2000 retuned items.  Helpful responses are appreciated.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...