How can I calculate in real-time the data rate (kbps) of events being indexed?

[hexx]

I would like to find out how much data per second (let's say in kilobytes per second) my indexer is receiving and indexing.

What search could help me find this out?

[mattlucas719]

better:
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/queues
| search title=parsingQueue* OR title=aggQueue* OR title=typingQueue* OR title=indexQueue*
| eval fill_perc=round(current_size_bytes / max_size_bytes * 100,2)
| fields splunk_server, title, fill_perc
| rex field=title "(?<queue_name>^\w+)(?:\.(?<pipeline_number>\d+))?"
| eval fill_perc = if(isnotnull(pipeline_number), "pset".pipeline_number.": ".fill_perc, fill_perc)
| chart values(fill_perc) over splunk_server by queue_name
| eval pset_count = mvcount(parsingQueue)
| join type=outer splunk_server [
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/indexer
| eval average_KBps = round(average_KBps, 0)
| eval status = if((reason == ".") OR (reason == "") OR isnull(reason), status, status.": ".reason)
| fields splunk_server, average_KBps, status]
| fields splunk_server pset_count average_KBps status parsingQueue aggQueue typingQueue indexQueue
| sort -average_KBps
| stats sum(average_KBps) AS "average_KBps"
| eval "average_MBs"=round(average_KBps/1024, 2)


[hexx · Accepted solution]

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.