Getting Data In

How can I calculate in real-time the data rate (kbps) of events being indexed?

Splunk Employee
Splunk Employee

I would like to find out how much data per second (let's say in kilobytes per second) my indexer is receiving and indexing.

What search could help me find this out?

Tags (3)
1 Solution

Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermicnoodle | eval searchtime=now() | eval secondselapsed=(time() - searchtime) | eval secs=if(secondselapsed<0,"1",secondselapsed) | eval esize=((len(raw)/1024)) | stats sum(esize) as sumesize, last(secs) AS seconds | stats last(sumesize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sumesize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

View solution in original post

Explorer

better:
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/queues
| search title=parsingQueue* OR title=aggQueue* OR title=typingQueue* OR title=indexQueue*
| eval fill_perc=round(current_size_bytes / max_size_bytes * 100,2)
| fields splunk_server, title, fill_perc
| rex field=title "(?<queue_name>^\w+)(?:\.(?<pipeline_number>\d+))?"
| eval fill_perc = if(isnotnull(pipeline_number), "pset".pipeline_number.": ".fill_perc, fill_perc)
| chart values(fill_perc) over splunk_server by queue_name
| eval pset_count = mvcount(parsingQueue)
| join type=outer splunk_server [
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/indexer
| eval average_KBps = round(average_KBps, 0)
| eval status = if((reason == ".") OR (reason == "") OR isnull(reason), status, status.": ".reason)
| fields splunk_server, average_KBps, status]
| fields splunk_server pset_count average_KBps status parsingQueue aggQueue typingQueue indexQueue
| sort -average_KBps
| stats sum(average_KBps) AS "average_KBps"
| eval "average_MBs"=round(average_KBps/1024, 2)

0 Karma

Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermicnoodle | eval searchtime=now() | eval secondselapsed=(time() - searchtime) | eval secs=if(secondselapsed<0,"1",secondselapsed) | eval esize=((len(raw)/1024)) | stats sum(esize) as sumesize, last(secs) AS seconds | stats last(sumesize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sumesize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

View solution in original post