Splunk Search

Dashboard panel statistic using extracted field not working

owulz
Explorer

Hi

I have a dashboard panel that displays (for a given server) 4 statistic values. Backups started, running, successful, failed.

The query section of code in my dashboard panel looks like this at the moment...

<query>
index=myindex sourcetype=bla:linux:syslog host="server.bla.COM" ABCEVENT
| stats
count(eval(searchmatch("BACKUP AND CPF1124"))) as "Started"
count(eval(searchmatch("BACKUP AND CPF1164 AND SUCCESS"))) as "Successful"
count(eval(searchmatch("BACKUP AND CPF1164 AND FAILURE"))) as "Failed"
| eval Running=if(Started-(Successful+Failed) &gt;= 0, Started-(Successful+Failed), 0)
| table Started, Running, Successful, Failed
</query>

The field Running is a calculated field which works, but not well as it relies on data that may not be unreliable. If time range is 24 hours it's not so bad, but if I view for past 7 days, there is increased chance that an event relating to backup started or backup completed successful or with failure may get missed and not ingested into Splunk. Seems to happen sometimes. Since field Running is not found through search; it's a calculated field, it relies on accuracy. The field calculation is used to imply if a backup is likely running.

I want to replace the value that is displayed for field Running with something like the following (based on new data I send to splunk). Idea is to fetch only the last occurence of this event from the past 5 minutes. The event returned will essentially include a count value that I want to extra and use in my panel as a statistic.

index=myindex sourcetype=bla:linux:syslog host="server.bla.COM" ABCEVENT *NONSBS earliest=-5m | eventstats max(_time) as maxtime | where _time=maxtime

When I do above as a regular Splunk search, I get a single event returned which is perfect.

I already created a field extraction which always shows up as an available field in my search results. It's called Jobs_Running.

What I would like to do is to replace this ...

| eval Running=if(Started-(Successful+Failed) &gt;= 0, Started-(Successful+Failed), 0)

With something similar to the above search string, adapted to work within the existing panel, so that I can display the new value for "Running" along side the existing fields "Started", "Successful" and "Failed".

Is there a way to do this?

One thing I'm not sure about is whether I can pull in the already extracted field (Jobs_Running) that ia visible when I do a regular search or do I need to perform a field extraction on the fly?

The expression is: ^(?:[^ \n]* ){9}(?P<Jobs_Running>\d+)

To summarise: Started, Successful, Failed are found from search over the time range and counted. Running is calculated on the fly. Now I want to pull in a value from a single (latest) occurence of an event searched from the last 5 minutes, and extract the field value.

Thanks

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...