topic Re: Daily index volume by sourcetype in Getting Data In

Daily index volume by sourcetype

mtanadsk — Mon, 22 Aug 2011 22:27:28 GMT

How does one go about calculating daily index volume by sourcetype?

I'm currently capturing all logged data and sending it to the main index, but moving away from that method, for performance/scalability reasons. We're going through a data classification exercise that will dictate our future state indexes, and as part of that exercise, want to get a better idea of the breakdown of data sizes, and thought sizing by 'sourcetype' would be a good starting point.

thanks for any information!

Regards,
-m

Re: Daily index volume by sourcetype

cnk — Mon, 28 Sep 2020 09:49:12 GMT

index=_internal source=metrics.log splunk_server="" | eval MB=kb/1024 | search group="per_sourcetype_thruput" | timechart span=1d sum(MB) by series

Re: Daily index volume by sourcetype

wbfoxii — Thu, 01 Nov 2012 19:38:20 GMT

In the Search app, hit the "status" dropdown and choose index activity. I was able to get index volume by sourcetype / server / at any time span. It was great for a built-in tool.

Running Splunk 4.3.2

Re: Daily index volume by sourcetype

gurinderbhatti — Thu, 10 Jul 2014 20:56:00 GMT

What if i want to know for a specific sourcetype in a specific index?
We have over 50+ indexes but for a couple of indexes, lets say index=a, index=b, index=c, i want to know how much data on a daily basis, my windows system/security/events logs are generating (min,max,avg) for a 30 day range.
Can someone write up a quick search for this? Much appreciated.

Re: Daily index volume by sourcetype

slierninja — Tue, 27 Jan 2015 22:40:11 GMT

This works with Splunk 6.2 for supporting GB by index...

index=_internal source=*metrics.log | eval GB=kb/(1024*1024) | search group="per_sourcetype_thruput" | timechart span=1d sum(GB) by series limit=20

Re: Daily index volume by sourcetype

kiran_mh — Wed, 08 Jun 2016 13:48:38 GMT

How can I get the index volume for a particular sourcetype, for example sourcetype=perfmon*..?

Re: Daily index volume by sourcetype

rashid47010 — Sun, 15 Apr 2018 18:32:44 GMT

I am new to splunk
I installed trial version which can index 500 MB/day. My question is that Domain controller can index how much events. Please just give me estimate.

Re: Daily index volume by sourcetype

ss026381 — Wed, 20 Jun 2018 13:57:24 GMT

Try this query, it will give you size in GB for each day.

index=_internal source=*metrics.log
series=wineventlog:security
| eval formatted_time=strftime(_time, "%x")
| Rename series as sourcetypes
| chart sum(kb) over sourcetypes by formatted_time
| sort - [ makeresults | addinfo | eval time="\"".strftime(info_max_time-1, "%x")."\"" | return $time]
| foreach */* [eval "<<FIELD>>"=round('<<FIELD>>'/1024/1024,2)." GB"]

Re: Daily index volume by sourcetype

ss026381 — Wed, 20 Jun 2018 14:33:47 GMT

Try below, select the time last 30 days in time picker.

 index=_internal source=*metrics.log group="per_sourcetype_thruput"
 [| metadata type=hosts ( index=a OR index=b OR index=c)| table host | format] 
 (series=wineventlog:system OR series=wineventlog:security OR series=wineventlog:application) 
 | eval formatted_time=strftime(_indextime, "%x")
 | Rename series as sourcetypes
 | chart sum(kb) over sourcetypes by formatted_time
 | sort - [ makeresults | addinfo | eval time="\"".strftime(info_max_time-1, "%x")."\"" | return $time]
 | foreach */* [eval "<<FIELD>>"=round('<<FIELD>>'/1024/1024,2)." GB"]

Re: Daily index volume by sourcetype

landen99 — Wed, 28 Nov 2018 20:58:28 GMT

I downvoted this post because incomplete. only top 10 sourcetypes recorded in metrics

Re: Daily index volume by sourcetype

landen99 — Wed, 28 Nov 2018 20:59:18 GMT

I downvoted this post because the results from this solution are incomplete. only the top 10 sourcetypes are recorded in metrics

Re: Daily index volume by sourcetype

rakesh44 — Sun, 03 Feb 2019 13:58:19 GMT

If i have 3 source type like type, item and payable then what would be command. Can you please give command for my sourctetype.Thanks

Re: Daily index volume by sourcetype

wyfwa4 — Fri, 21 May 2021 10:23:56 GMT

By default the metrics will only provide details on the top 10 items in a series - this applies to almost all the metrics that are collected.

https://docs.splunk.com/Documentation/Splunk/8.1.3/Troubleshooting/Aboutmetricslog

The way to address this, is to change the limit applied using the limits.conf file - the following stanza added to a indexer / HF will increase the number of items which are tracked

[metrics] maxseries = 50 interval = 60

This will ensure the top 50 items are included - so you can adjust this to reflect the likely number of sourcetypes if that is the main metric to be tracked

The "interval" reference allows you to change the frequency that this data is collected to control the volume of data generated in the log. In the same, the frequency is 60seconds to allow minute by minute tracking - but that may be excessive for some purposes.