I'm trying to create some logic within my search, and it requires some IF THEN AND logic, which I know Splunk has the capability to do, but I don't know how to make it work the way I'm needing it.
I have 2 different types of machines I'm searching, and I'm trying to alert on two distinct values.
example: if machines named host10* have a mount with mount=/boot, AND have drive space over 90% then alert, AND if machines named host20* have a mount with mount=/boot AND drive space over 95% alert.
Working Query:
index=nix sourcetype=df host=myHost10 * OR host=myHost20*
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn
| where pctUsed > 90
| sort - pctUsed
I thought about using |eval field=if(coalesce...)
but I don't think it fits my needs here, as both host types will have a value, it's just that the value needs to be filtered differently based upon the system type. maybe a subsearch?
Any help would be appreciated.
I tried all of these, but alas... there was no success. I couldn't get the filter to populate all the results I needed. In the end I simply broke this into two alerts, 1 with filters around the mount and the pctUsed, and the other around everything EXCEPT that mount, but still with the pctUsed.
I tried all of these, but alas... there was no success. I couldn't get the filter to populate all the results I needed. In the end I simply broke this into two alerts, 1 with filters around the mount and the pctUsed, and the other around everything EXCEPT that mount, but still with the pctUsed.
Hi there @tmarlette
Try something like this.
index=nix sourcetype=df host=myHost10 * OR host=myHost20*
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn
| eval condition=case(host="host10*" AND MountedOn="/boot" AND pctUsed> 90, "1", machine="host20*" AND MountedOn="/boot" AND pctUsed > 95, "1", 1==1, "0")
| sort - pctUsed
| where condition=="1"
Hope it helps.
Try this query.
index=nix sourcetype=df (host=myHost10 * OR host=myHost20*) mount="/boot"
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn
| search (host=myHost10* pctUsed > 90) OR (host=myhost20* pctUsed>95)
| sort - pctUsed
This would work, if I only had to check the /boot mount, but I need to show the results of all other mounts on the system as well. =(
Try like this
index=nix sourcetype=df host=myHost10 * OR host=myHost20*
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn
| where (like(host,"myHost10%) AND pctUsed > 90) OR (like(host,"myHost20%) AND pctUsed > 95)
| sort - pctUsed
This seems so close. I just have to choose a mount on my linux systems, otherwise the data is pointless. This is what I have
index=nix sourcetype=df host=myHost10 * OR host=myHost20*
| stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn
| where (like(MountedOn,"%") AND pctUsed > 90) OR (like(MountedOn,"home/work%") AND pctUsed > 95)
| sort - pctUsed
it's showing me weird results though. It's only showing me the machines that have 'home/work' mount ABOVE 90%.
So this is my filter now, I hope this explains what I'm trying to better:
index=nix sourcetype=df host=myHost10 * OR host=myHost20* | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn | where (like(MountedOn,"%") AND pctUsed > 50 AND MountedOn!="/home/work*") OR (like(MountedOn,"home/work%") AND pctUsed > 95) | sort - pctUsed
I'm looking for all mounts above 50%, and then any /home/work
mounts that are above 95%, but I need them all displayed in a single table. The above query is still returning results with the /home/work below 95%.
here is an image
Give this a try
index=nix sourcetype=df host=myHost10* OR host=myHost20* | stats first(PercentUsedSpace) as pctUsed latest(Avail) as Avail by host, Filesystem, filesystem_type, Size, Used, MountedOn | where (like(MountedOn,"home/work%") AND pctUsed > 95) OR (NOT like(MountedOn,"home/work%") AND pctUsed > 50 ) | sort - pctUsed
still got the same result set. =(