Forgive my ignorance as I'm relatively new to Splunk. I'm currently hitting what I *think* is a data type issue, but I'm not quite sure how to proceed. We are using the Splunk add-on for Unix and Linux to return the set of 'df-metric' values. I would like to set up a simple alert on the metric_name:df_metric.UsePct value, alerting when the value exceeds 85%. I'm able to run this query and return data using an equality operator on that value:
index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=8
...however I'm NOT able to return data when perform an 'greater than' comparison on the metric_name:df_metric.UsePct value like this:
index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct">8
Initially I tried manipulating the metric_name:df_metric.UsePct with the tonumber() function, thinking I was possibly receiving a string back, however that does not result in the data I would expect to see.
If anyone has guidance on traversing the data set returned by df_metric or any other points, I would appreciate it!
Thank you!
NOTE: I'm using 8 as a value for the metric_name:df_metric.UsePct only for testing purposes. This will, of course, need to be adjusted to 85 for the live alert.
Alright!
It looks like the pct value is a multi value field for some reason. Probably the values are of metrics and you are not using a metrics index.
You can confirm this by
index= sourcetype="df_metric"
|eval mc=mvcount('metric_name:df_metric.UsePct')
|table metric_name*,mc
Nevertheless,just try this and see if it produces any result
your search
|eval df_pct=mvindex('metric_name:df_metric.UsePct',0)
|where df_pct > 8
Try using where or search
e.g.
index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=*|search "metric_name:df_metric.UsePct" > 8
Thank you and yes, that makes sense to me as well, however that yields a type error:
Error in 'where' command: Type checking failed. The '>' operator received different types.
I also tried wrapping the 'df_metric.UsePct' value in the 'where' clause in a 'tonumber()' call to force a type conversion, however that does not seem to yield anything.
Yes, sorry I didnt pay attention to the fieldname. There error is because, we are comparing a "string" with a number. As mention, you may 'search' instead - I updated the answer
So two options
"your current search" |search "metric_name:df_metric.UsePct" > 8
or
"your current search"|rename "metric_name:df_metric.UsePct" as df_pct |where df_pct > 8
It appears both 'search' and 'where' result in a string type. For example, when I perform this search:
index="linuxlogs" sourcetype="df_metric" host="<redacted>" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" = 8
I am able to retrieve results, I assume because the search "metric_name:df_metrci.UsePct" = 8 is correctly performing an equality test on the 8 as a string value:
However when I change this to the ">" operator and value, it does not return the result above, which should meet this search criteria:
index="linuxlogs" sourcetype="df_metric" host="<redacted>" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" > 0
Do you have any suggestions to on how to force a type conversion on the value being returned in "metric_name:df_metric.UsePct"?
Alright!
It looks like the pct value is a multi value field for some reason. Probably the values are of metrics and you are not using a metrics index.
You can confirm this by
index= sourcetype="df_metric"
|eval mc=mvcount('metric_name:df_metric.UsePct')
|table metric_name*,mc
Nevertheless,just try this and see if it produces any result
your search
|eval df_pct=mvindex('metric_name:df_metric.UsePct',0)
|where df_pct > 8
Perfect! You are right - it is an index and that solution worked. Thank you very much!
In hindsight, the UI was trying to tell me that with the multiple values appearing in the 'df_metric.UsePct' field in the screenshot I posted above...I should have seen that.
I'm saving both of these queries off for future use - thank you again @renjith_nair !