Getting Data In

How can I calculate in real-time the data rate (kbps) of events being indexed?

hexx
Splunk Employee
Splunk Employee

I would like to find out how much data per second (let's say in kilobytes per second) my indexer is receiving and indexing.

What search could help me find this out?

Tags (3)
1 Solution

hexx
Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

View solution in original post

mattlucas719
Explorer

better:
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/queues
| search title=parsingQueue* OR title=aggQueue* OR title=typingQueue* OR title=indexQueue*
| eval fill_perc=round(current_size_bytes / max_size_bytes * 100,2)
| fields splunk_server, title, fill_perc
| rex field=title "(?<queue_name>^\w+)(?:\.(?<pipeline_number>\d+))?"
| eval fill_perc = if(isnotnull(pipeline_number), "pset".pipeline_number.": ".fill_perc, fill_perc)
| chart values(fill_perc) over splunk_server by queue_name
| eval pset_count = mvcount(parsingQueue)
| join type=outer splunk_server [
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/indexer
| eval average_KBps = round(average_KBps, 0)
| eval status = if((reason == ".") OR (reason == "") OR isnull(reason), status, status.": ".reason)
| fields splunk_server, average_KBps, status]
| fields splunk_server pset_count average_KBps status parsingQueue aggQueue typingQueue indexQueue
| sort -average_KBps
| stats sum(average_KBps) AS "average_KBps"
| eval "average_MBs"=round(average_KBps/1024, 2)

0 Karma

hexx
Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...