topic Re: How do I use IF / THEN / AND in logic in a search? in Splunk Search

How do I use IF / THEN / AND in logic in a search?

tmarlette — Thu, 17 Aug 2017 17:03:42 GMT

I'm trying to create some logic within my search, and it requires some IF THEN AND logic, which I know Splunk has the capability to do, but I don't know how to make it work the way I'm needing it.

I have 2 different types of machines I'm searching, and I'm trying to alert on two distinct values.

example: if machines named host10* have a mount with mount=/boot, AND have drive space over 90% then alert, AND if machines named host20* have a mount with mount=/boot AND drive space over 95% alert.

Working Query:

index=nix sourcetype=df host=myHost10 * OR host=myHost20* 
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn 
| where pctUsed > 90
| sort - pctUsed

I thought about using |eval field=if(coalesce...) but I don't think it fits my needs here, as both host types will have a value, it's just that the value needs to be filtered differently based upon the system type. maybe a subsearch?

Any help would be appreciated.

Re: How do I use IF / THEN / AND in logic in a search?

somesoni2 — Thu, 17 Aug 2017 17:24:36 GMT

Try like this

 index=nix sourcetype=df host=myHost10 * OR host=myHost20* 
 | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn 
 | where (like(host,"myHost10%) AND pctUsed > 90) OR (like(host,"myHost20%) AND pctUsed > 95) 
 | sort - pctUsed

Re: How do I use IF / THEN / AND in logic in a search?

richgalloway — Thu, 17 Aug 2017 17:26:58 GMT

Try this query.

 index=nix sourcetype=df (host=myHost10 * OR host=myHost20*) mount="/boot"
 | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn 
 | search (host=myHost10* pctUsed > 90) OR (host=myhost20* pctUsed>95)
 | sort - pctUsed

Re: How do I use IF / THEN / AND in logic in a search?

alemarzu — Thu, 17 Aug 2017 17:30:34 GMT

Hi there @tmarlette

Try something like this.

index=nix sourcetype=df host=myHost10 * OR host=myHost20* 
 | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn 
 | eval condition=case(host="host10*" AND MountedOn="/boot" AND pctUsed> 90, "1", machine="host20*" AND MountedOn="/boot" AND pctUsed > 95, "1", 1==1, "0")
 | sort - pctUsed 
 | where condition=="1"

Hope it helps.

Re: How do I use IF / THEN / AND in logic in a search?

tmarlette — Thu, 17 Aug 2017 17:50:22 GMT

This seems so close. I just have to choose a mount on my linux systems, otherwise the data is pointless. This is what I have

 index=nix sourcetype=df host=myHost10 * OR host=myHost20* 
 | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn 
| where (like(MountedOn,"%") AND pctUsed > 90) OR (like(MountedOn,"home/work%") AND pctUsed > 95)
 | sort - pctUsed

it's showing me weird results though. It's only showing me the machines that have 'home/work' mount ABOVE 90%.

Update, the last statement was my own stupidity, please ignore that. 🙂 It's just not filtering out the >95%. I think my filter is too vauge.

Re: How do I use IF / THEN / AND in logic in a search?

tmarlette — Thu, 17 Aug 2017 17:51:37 GMT

This would work, if I only had to check the /boot mount, but I need to show the results of all other mounts on the system as well. =(

Re: How do I use IF / THEN / AND in logic in a search?

tmarlette — Thu, 17 Aug 2017 18:16:55 GMT

So this is my filter now, I hope this explains what I'm trying to better:

  index=nix sourcetype=df host=myHost10 * OR host=myHost20*    | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn  | where (like(MountedOn,"%") AND pctUsed > 50 AND MountedOn!="/home/work*") OR (like(MountedOn,"home/work%") AND pctUsed > 95)   | sort - pctUsed

I'm looking for all mounts above 50%, and then any /home/work mounts that are above 95%, but I need them all displayed in a single table. The above query is still returning results with the /home/work below 95%.

here is an image

Re: How do I use IF / THEN / AND in logic in a search?

somesoni2 — Thu, 17 Aug 2017 18:40:50 GMT

Give this a try

index=nix sourcetype=df host=myHost10* OR host=myHost20*    | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn  | where (like(MountedOn,"home/work%") AND pctUsed > 95) OR   (NOT like(MountedOn,"home/work%") AND pctUsed > 50 ) | sort - pctUsed

Re: How do I use IF / THEN / AND in logic in a search?

tmarlette — Thu, 17 Aug 2017 18:45:19 GMT

still got the same result set. =(

Re: How do I use IF / THEN / AND in logic in a search?

tmarlette — Tue, 05 Sep 2017 22:10:02 GMT

I tried all of these, but alas... there was no success. I couldn't get the filter to populate all the results I needed. In the end I simply broke this into two alerts, 1 with filters around the mount and the pctUsed, and the other around everything EXCEPT that mount, but still with the pctUsed.