topic Re: How to determine the indexing volume by source in Getting Data In

How to determine the indexing volume by source

laurent_ — Tue, 06 May 2014 11:52:42 GMT

On an 'All time' range, the two following searches provide different results. The first one gives the expected result (exhaustive) while the second one is incomplete.

index="my_index" | stats values(source)

index="_internal" source="*metrics.log" group="per_source_thruput" series="/path/to/raw/data/*" | stats by series | fields series

We are trying to get the indexing volume by source, according to http://answers.splunk.com/answers/140/how-do-i-determine-my-indexing-volume-by-host-source-or-sourcetype

index="_internal" source="*metrics.log" group="per_source_thruput" series="/path/to/raw/data/*" | chart sum(kb) by series | sort - sum(kb)

But a lot of sources are missing, any idea ?

Re: How to determine the indexing volume by source

MuS — Tue, 06 May 2014 12:01:59 GMT

Hi laurent_,

by default metrics.log only reports on the top 10 results for each type.
You can change this in metrics stanza of limits.conf

Update:

here are some searches that does not use metrics.log

per source:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by s useother=false

summary per day per pool for the previous days:

index=_internal source=*license_usage* type=RolloverSummary | bucket _time span=1d | stats sum(b) AS volume by _time pool

per pool:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool

per sourcetype:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by st useother=false

per host:

 index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by h useother=false

per indexer:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by i useother=false

hope this helps ...

cheers, MuS

Re: How to determine the indexing volume by source

laurent_ — Mon, 28 Sep 2020 16:34:16 GMT

Thank you for your answer MuS.

If we use the index=internal source=*license_usage.log type=Usage we also get the top 10 sources and not _all sources.

We have not found the value allowing to log all sources in *metrics.log and had to set arbitrarily the parameter in limits.conf
[metrics]
maxseries = 100000

Any help ?

Thanks in advance

Re: How to determine the indexing volume by source

MuS — Fri, 09 May 2014 09:44:37 GMT

according to the docs http://docs.splunk.com/Documentation/Splunk/6.0.3/Troubleshooting/WhatSplunklogsaboutitself license_usage.log contains all information, not only top 10.
limits.conf is the place for the change, like I told you. Did you restart Splunk afterwards? Also this will only be valid for new events.

Re: How to determine the indexing volume by source

MuS — Fri, 09 May 2014 14:11:03 GMT

please mark this as answered by ticking the tick - thx 😉