Splunk Search

How to get largest numeric (using regex) value displayed on a dashboard from a particular search?

agoktas
Communicator

We have a search that comes back with the following raw log data in every few lines:

getUnitofWorkThread-1 incrementing totalRecordsProcessedCount to 85441

That number increments and I want to display the latest number with the search range (i.e.: If I have 20 values (number increments), I only need the latest/largest number).

How do I do this?

I'm open to any suggestions.

Should I shorten the range to the last 1 minute? Real-time searches cause unnecessary load on the search head, so I was hoping to do perhaps the last 5 minutes and perform the search every 5 minutes.

Additionally, I want to create a dial/gauge that shows x number of log entries (realtime search is ok in this case) for past x seconds/minutes. So any suggestions on how to create that would be appreciated.

The goal is to ensure that we don't have any delays in processing a job that will take about 48 hours. If there is a hang and the output to log slows down dramatically, or even stops, we want an alert. But first I wanted to get a handy dashboard going.

Thanks!

0 Karma
1 Solution

somesoni2
Revered Legend

You can try like this to get the largest number. Shortening the timerange would help in query performance.

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber

View solution in original post

muebel
SplunkTrust
SplunkTrust

As pointed out in other comments/answers, using stats with the max function, or sorting descending with a head of 1 on the extracted field would give you the largest number for a given timerange.

For the count of total events within a window, you could do something like:

sourcetype=processing_job | stats count as "Total Events"

for some given realtime window

0 Karma

agoktas
Communicator

Thanks for that info! 🙂

I was hoping to add this to a Radial Gauge, but default is 100.

We expect about 5,000 log entries/lines per hour and I am curious how we can change the Radial Gauge threshold/value.

Also, how can we revert the direction? I.e.: 5,000 entries is in the green and 0 would be red.

Thanks.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Stats max is great way.. You could also Sort, and head?

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | sort - SomeNumber | head 1
0 Karma

somesoni2
Revered Legend

You can try like this to get the largest number. Shortening the timerange would help in query performance.

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber

agoktas
Communicator

Thanks for the answer. That worked perfectly!

One last question,

How would I change the color of the value's font when I have this in a dashboard as a "Single Value" within the panel?

Currently is default bold + black font.

Thanks!

0 Karma

somesoni2
Revered Legend

Have a look at the Dashboard example app
https://splunkbase.splunk.com/app/1603/

There are examples for Single Value Elements which can help you get these kind of customizations.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...