We have a customer that is using the EMC Isilon App for Splunk to collect data from several Isilon clusters and overall they are very happy with the results. However, we've begun getting requests for some customized reports using the data collected by the app - primarily focused on % utilization, bytes available, and bytes free. Looking at the app itself, it already reports on all 3 of these data points, but it does so individually and our customers would like to have them combined into a single graph. Usually this wouldn't be an issue, except that the Isilon app stores these events as nested JSON entries, using the same field names to represent multiple pieces of data. For example, the stats.key field contains numerous entries for ifs.percent.used, ifs.bytes.avail, and ifs.bytes.used. The actual values for any of these keys are stored in the stats.value field. The problem is that just from looking at the stats.value field, there's no way to tell what key the value relates back to. I can create the queries easily enough to look at any single data point - the problem I'm having is when trying to create as single search that contains all three data points.
Figuring that we needed to use a subsearch of some sort in order to represent each data point, the closest I've been able to come is:
index=isilon Cluster_Name=cluster1 |eval percent_used=[search index=isilon Cluster_Name=cluster1 stats.key = "ifs.percent.used" |stats max(stats.value) by _time] |eval bytes_used=[search index=isilon Cluster_Name=cluster1 stats.key = "ifs.bytes.used" |stats max(stats.value) by _time] |eval bytes_available=[search index=isilon Cluster_Name=cluster1 stats.key = "ifs.bytes.avail" |stats max(stats.value) by _time] |timechart max(percent_used) max(bytes_used) max(bytes_available)
Unfortunately I keep getting an error in the eval command that the fields cannot be assigned a boolean result.
I'm stuck on this one - any help would be greatly appreciated. If you've not used the EMC App or are not familiar with its data and a sample of actual events would help, I'll be happy to provide...
| makeresults | eval data="%{\"stats\": {\"value\": 44.85097033572893, \"key\": \"ifs.percent.used\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:10\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}
%{\"stats\": {\"value\": 526733856178176, \"key\": \"ifs.bytes.free\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:09\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}
%{\"stats\": {\"value\": 518188251906048, \"key\": \"ifs.bytes.avail\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:09\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}" | rex max_match=9999 field=data "(?<json>[^%]+)" | mvexpand json | table json | spath input=json | rename stats.* AS *| eval used=if(key = "ifs.percent.used", value, null()) | eval free=if(key = "ifs.bytes.free", value, null()) | eval avail=if(key = "ifs.bytes.avail", value, null()) | stats last(avail) as avail last(free) as free last(used) as used by node
| makeresults | eval data="%{\"stats\": {\"value\": 44.85097033572893, \"key\": \"ifs.percent.used\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:10\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}
%{\"stats\": {\"value\": 526733856178176, \"key\": \"ifs.bytes.free\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:09\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}
%{\"stats\": {\"value\": 518188251906048, \"key\": \"ifs.bytes.avail\", \"devid\": 0, \"error_code\": null, \"time\": 1489611549, \"error\": null}, \"timestamp\": \"2017-03-15 20:59:09\", \"node\": \"10.192.14.118\", \"namespace\": \"statistics\"}" | rex max_match=9999 field=data "(?<json>[^%]+)" | mvexpand json | table json | spath input=json | rename stats.* AS *| eval used=if(key = "ifs.percent.used", value, null()) | eval free=if(key = "ifs.bytes.free", value, null()) | eval avail=if(key = "ifs.bytes.avail", value, null()) | stats last(avail) as avail last(free) as free last(used) as used by node
Here's a sample event for each type:
Bytes Available
{
namespace: statistics
node: 1.1.1.1
stats: {
devid: 0
error: null
error_code: null
key: ifs.bytes.avail
time: 1487863543
value: 1321045700624384
}
timestamp: 2017-02-23 10:30:24
}
Bytes Free
{
namespace: statistics
node: 1.1.1.1
stats: {
devid: 0
error: null
error_code: null
key: ifs.bytes.free
time: 1487863543
value: 1321045700000573
}
timestamp: 2017-02-23 10:30:24
}
Percent Used
{
namespace: statistics
node: 1.1.1.1
stats: {
devid: 0
error: null
error_code: null
key: ifs.percent.used
time: 1487863543
value: 43.754521
}
timestamp: 2017-02-23 10:30:24
}
Show us a few representative raw events.