Getting Data In

Combine multiple index searches into one overall stat

danroberts
Explorer

Hello, 

I'm trying to create a RAG dashboard that will show different colours should an issue occur with a service e.g. if a service stops working the stat would show as one and the colour would turn red, I can do this but what I am struggling with is combining multiple index searches into one overall stat e.g. index "windows_perfmon" disk runs out of space, stat increases to 1, a winhostmon index service stops and that stat increases to one, I'm struggling to combine these into one overall stat which would be 2 in this example. 

The current search I am using is: 

(index=winhostmon host="Splunktest" "Type=Service" sourcetype=WinHostMon DisplayName="Print Spooler" OR DisplayName="Snow Inventory Agent" StartMode="Auto" State="Stopped") OR
(index="windows_perfmon" host="Splunktest" object="LogicalDisk" counter="% Free Space" OR counter="Free Megabytes")
| eval diskInfoA = if(counter=="% Free Space",mvzip(instance,Value),null())
| eval diskInfoA1 = if(isnotnull(diskInfoA),mvzip(diskInfoA,counter),null())
| eval diskInfoB = if(counter=="Free Megabytes",mvzip(instance,Value),null())
| eval diskInfoB1 = if(isnotnull(diskInfoB),mvzip(diskInfoB,counter),null())
| stats list(diskInfoA1) AS "diskInfoA1", list(diskInfoB1) AS "diskInfoB1" by host, instance, _time
| makemv diskInfoA1 delim=","
| makemv diskInfoB1 delim=","
| eval freePerc = mvindex(diskInfoA1,1)
| eval freeMB = mvindex(diskInfoB1,1)
| eval usage=round(100-freePerc,2)
| eval GB = round(freeMB/1024,2)
| eval totalDiskGB = GB/(freePerc/100)
| stats max(usage) AS "Disk Usage", max(GB) AS "Disk Free", max(totalDiskGB) AS "Total Disk Size (GB)" by host instance
| where not instance="_Total"
| where NOT LIKE(instance,"%Hard%")
| search "Disk Usage" >90
| stats count

The result I get is just count=1 

Note in the above example I have stopped the print spooler on the server so the event count should be 2 as there is a disk that is also running above 90%

I have also tried the append version but again I cannot get it to combine the results.

index=winhostmon host="Splunktest" "Type=Service" sourcetype=WinHostMon DisplayName="Print Spooler" OR DisplayName="Snow Inventory Agent" StartMode="Auto" State="Stopped" | stats count|rename count as Service

|append [ search index="windows_perfmon" host="Splunktest" object="LogicalDisk" counter="% Free Space" OR counter="Free Megabytes"

| eval diskInfoA = if(counter=="% Free Space",mvzip(instance,Value),null())

| eval diskInfoA1 = if(isnotnull(diskInfoA),mvzip(diskInfoA,counter),null())

| eval diskInfoB = if(counter=="Free Megabytes",mvzip(instance,Value),null())

| eval diskInfoB1 = if(isnotnull(diskInfoB),mvzip(diskInfoB,counter),null())

| stats list(diskInfoA1) AS "diskInfoA1", list(diskInfoB1) AS "diskInfoB1" by host, instance, _time

| makemv diskInfoA1 delim=","

| makemv diskInfoB1 delim=","

| eval freePerc = mvindex(diskInfoA1,1)

| eval freeMB = mvindex(diskInfoB1,1)

| eval usage=round(100-freePerc,2)

| eval GB = round(freeMB/1024,2)

| eval totalDiskGB = GB/(freePerc/100)

| stats max(usage) AS "Disk Usage", max(GB) AS "Disk Free", max(totalDiskGB) AS "Total Disk Size (GB)" by host instance

| where not instance="_Total"

| where NOT LIKE(instance,"%Hard%")

| search "Disk Usage" >90

| stats count|rename count as Disk

]

The end goal of this is to just show one stat on a dashboard and when you click on that number it opens another dashboard that shows you the detail. 

 

Any help would be appreciated. 

Labels (2)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Rounding off the Splunk Dashboard Contest

What does a contest-winning Splunk dashboard look like? In this case, it isn't in a browser tab at all. It ...

A Four Part Event Series: AI + Observability: AI Agents, LLMs, Apps, & Infrastructure

AI + Observability: AI Agents, LLMs, Apps, & Infrastructure The rapid evolution of artificial intelligence ...

Splunk Technical Support Is Moving to Cisco Support Tools

Introduction Splunk technical support is transitioning to Cisco’s support environment. This change brings ...