topic How can I calculate in real-time the data rate (kbps) of events being indexed? in Getting Data In

How can I calculate in real-time the data rate (kbps) of events being indexed?

hexx — Fri, 23 Sep 2011 15:45:51 GMT

I would like to find out how much data per second (let's say in kilobytes per second) my indexer is receiving and indexing.

What search could help me find this out?

Re: How can I calculate in real-time the data rate (kbps) of events being indexed?

hexx — Mon, 28 Sep 2020 09:54:14 GMT

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

Re: How can I calculate in real-time the data rate (kbps) of events being indexed?

mattlucas719 — Wed, 29 Mar 2017 16:20:49 GMT

better:
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/queues | search title=parsingQueue* OR title=aggQueue* OR title=typingQueue* OR title=indexQueue* | eval fill_perc=round(current_size_bytes / max_size_bytes * 100,2) | fields splunk_server, title, fill_perc | rex field=title "(?<queue_name>^\w+)(?:\.(?<pipeline_number>\d+))?" | eval fill_perc = if(isnotnull(pipeline_number), "pset".pipeline_number.": ".fill_perc, fill_perc) | chart values(fill_perc) over splunk_server by queue_name | eval pset_count = mvcount(parsingQueue) | join type=outer splunk_server [ | rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/indexer | eval average_KBps = round(average_KBps, 0) | eval status = if((reason == ".") OR (reason == "") OR isnull(reason), status, status.": ".reason) | fields splunk_server, average_KBps, status] | fields splunk_server pset_count average_KBps status parsingQueue aggQueue typingQueue indexQueue | sort -average_KBps | stats sum(average_KBps) AS "average_KBps" | eval "average_MBs"=round(average_KBps/1024, 2)