Splunk Search

How to write a search to identify the total amount of data is being indexed by MB per minute?

danielsofoulis
Path Finder

I need to identify the total amount of data is being indexed by my indexer cluster, by MB per minute.
I think the best way would be to search against the license.log file but I am having trouble with writing the search.
Any assistance is greatly appreciated.

1 Solution

twinspop
Influencer

Try this, adding indexers as required:

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1m sum(eval(kb/1024)) as "MB/min"

If you run for more than a few hours, forcing a different span in the timechart, you'll need to do some extra math. This one has 5 minute buckets and therefore requires dividing the value by 5.

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=5m sum(eval(kb/1024/5)) as "MB/min"

This one by hour

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1h sum(eval(kb/1024/60)) as "MB/min"

View solution in original post

Richfez
SplunkTrust
SplunkTrust

You could start from the DMC's searches. (Assuming you are on a reasonably recent Splunk version).

Settings, Distributed Management Console. Then in the DMC's menus, drop down Indexing, then look at the various "Indexing Performance" pages.

If any of those is close, you can hover your mouse over the chart and underneath it will be a magnifying glass icon to "Open in Search".

While it may or may not be exactly what you need, it should get you in the ballpark. If you need more help after finding those and looking at them, ask away!

Happy Splunking!
Rich

0 Karma

twinspop
Influencer

Try this, adding indexers as required:

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1m sum(eval(kb/1024)) as "MB/min"

If you run for more than a few hours, forcing a different span in the timechart, you'll need to do some extra math. This one has 5 minute buckets and therefore requires dividing the value by 5.

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=5m sum(eval(kb/1024/5)) as "MB/min"

This one by hour

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1h sum(eval(kb/1024/60)) as "MB/min"

danielsofoulis
Path Finder

Thank up for your help.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...