We have a search that comes back with the following raw log data in every few lines:
getUnitofWorkThread-1 incrementing totalRecordsProcessedCount to 85441
That number increments and I want to display the latest number with the search range (i.e.: If I have 20 values (number increments), I only need the latest/largest number).
How do I do this?
I'm open to any suggestions.
Should I shorten the range to the last 1 minute? Real-time searches cause unnecessary load on the search head, so I was hoping to do perhaps the last 5 minutes and perform the search every 5 minutes.
Additionally, I want to create a dial/gauge that shows x number of log entries (realtime search is ok in this case) for past x seconds/minutes. So any suggestions on how to create that would be appreciated.
The goal is to ensure that we don't have any delays in processing a job that will take about 48 hours. If there is a hang and the output to log slows down dramatically, or even stops, we want an alert. But first I wanted to get a handy dashboard going.
Thanks!
You can try like this to get the largest number. Shortening the timerange would help in query performance.
index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber
As pointed out in other comments/answers, using stats with the max function, or sorting descending with a head of 1 on the extracted field would give you the largest number for a given timerange.
For the count of total events within a window, you could do something like:
sourcetype=processing_job | stats count as "Total Events"
for some given realtime window
Thanks for that info! 🙂
I was hoping to add this to a Radial Gauge, but default is 100.
We expect about 5,000 log entries/lines per hour and I am curious how we can change the Radial Gauge threshold/value.
Also, how can we revert the direction? I.e.: 5,000 entries is in the green and 0 would be red.
Thanks.
Stats max is great way.. You could also Sort, and head?
index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | sort - SomeNumber | head 1
You can try like this to get the largest number. Shortening the timerange would help in query performance.
index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber
Thanks for the answer. That worked perfectly!
One last question,
How would I change the color of the value's font when I have this in a dashboard as a "Single Value" within the panel?
Currently is default bold + black font.
Thanks!
Have a look at the Dashboard example app
https://splunkbase.splunk.com/app/1603/
There are examples for Single Value Elements which can help you get these kind of customizations.