How to write a search to identify the total amount of data is being indexed by MB per minute?

[danielsofoulis]

I need to identify the total amount of data is being indexed by my indexer cluster, by MB per minute.
I think the best way would be to search against the license.log file but I am having trouble with writing the search.
Any assistance is greatly appreciated.

[Richfez]

You could start from the DMC's searches. (Assuming you are on a reasonably recent Splunk version).

Settings, Distributed Management Console. Then in the DMC's menus, drop down Indexing, then look at the various "Indexing Performance" pages.

If any of those is close, you can hover your mouse over the chart and underneath it will be a magnifying glass icon to "Open in Search".

While it may or may not be exactly what you need, it should get you in the ballpark. If you need more help after finding those and looking at them, ask away!

Happy Splunking!
Rich

[twinspop · Accepted solution]

Try this, adding indexers as required:

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1m sum(eval(kb/1024)) as "MB/min"

If you run for more than a few hours, forcing a different span in the timechart, you'll need to do some extra math. This one has 5 minute buckets and therefore requires dividing the value by 5.

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=5m sum(eval(kb/1024/5)) as "MB/min"

This one by hour

(host=your_indexer OR host=your_indexer2 OR host=your_indexer3) index=_internal metrics name=index_thruput sourcetype=splunkd | 
    timechart span=1h sum(eval(kb/1024/60)) as "MB/min"

[danielsofoulis]

Thank up for your help.