Splunk Search

How to get largest numeric (using regex) value displayed on a dashboard from a particular search?

agoktas
Communicator

We have a search that comes back with the following raw log data in every few lines:

getUnitofWorkThread-1 incrementing totalRecordsProcessedCount to 85441

That number increments and I want to display the latest number with the search range (i.e.: If I have 20 values (number increments), I only need the latest/largest number).

How do I do this?

I'm open to any suggestions.

Should I shorten the range to the last 1 minute? Real-time searches cause unnecessary load on the search head, so I was hoping to do perhaps the last 5 minutes and perform the search every 5 minutes.

Additionally, I want to create a dial/gauge that shows x number of log entries (realtime search is ok in this case) for past x seconds/minutes. So any suggestions on how to create that would be appreciated.

The goal is to ensure that we don't have any delays in processing a job that will take about 48 hours. If there is a hang and the output to log slows down dramatically, or even stops, we want an alert. But first I wanted to get a handy dashboard going.

Thanks!

0 Karma
1 Solution

somesoni2
Revered Legend

You can try like this to get the largest number. Shortening the timerange would help in query performance.

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber

View solution in original post

muebel
SplunkTrust
SplunkTrust

As pointed out in other comments/answers, using stats with the max function, or sorting descending with a head of 1 on the extracted field would give you the largest number for a given timerange.

For the count of total events within a window, you could do something like:

sourcetype=processing_job | stats count as "Total Events"

for some given realtime window

0 Karma

agoktas
Communicator

Thanks for that info! 🙂

I was hoping to add this to a Radial Gauge, but default is 100.

We expect about 5,000 log entries/lines per hour and I am curious how we can change the Radial Gauge threshold/value.

Also, how can we revert the direction? I.e.: 5,000 entries is in the green and 0 would be red.

Thanks.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Stats max is great way.. You could also Sort, and head?

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | sort - SomeNumber | head 1
0 Karma

somesoni2
Revered Legend

You can try like this to get the largest number. Shortening the timerange would help in query performance.

index=_internal sourcetype=splunkd | rex "(?<SomeNumber>\d+)$" | stats max(SomeNumber) as SomeNumber

agoktas
Communicator

Thanks for the answer. That worked perfectly!

One last question,

How would I change the color of the value's font when I have this in a dashboard as a "Single Value" within the panel?

Currently is default bold + black font.

Thanks!

0 Karma

somesoni2
Revered Legend

Have a look at the Dashboard example app
https://splunkbase.splunk.com/app/1603/

There are examples for Single Value Elements which can help you get these kind of customizations.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...