Getting Data In

How to determine the indexing volume by source

Explorer

On an 'All time' range, the two following searches provide different results. The first one gives the expected result (exhaustive) while the second one is incomplete.

index="my_index" | stats values(source)

index="_internal" source="*metrics.log" group="per_source_thruput" series="/path/to/raw/data/*" | stats by series | fields series

We are trying to get the indexing volume by source, according to http://answers.splunk.com/answers/140/how-do-i-determine-my-indexing-volume-by-host-source-or-source...

index="_internal" source="*metrics.log" group="per_source_thruput" series="/path/to/raw/data/*" | chart sum(kb) by series | sort - sum(kb)

But a lot of sources are missing, any idea ?

Tags (1)

SplunkTrust
SplunkTrust

Hi laurent_,

by default metrics.log only reports on the top 10 results for each type.
You can change this in metrics stanza of limits.conf

Update:

here are some searches that does not use metrics.log

per source:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by s useother=false

summary per day per pool for the previous days:

index=_internal source=*license_usage* type=RolloverSummary | bucket _time span=1d | stats sum(b) AS volume by _time pool

per pool:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool

per sourcetype:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by st useother=false

per host:

 index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by h useother=false

per indexer:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by i useother=false 

hope this helps ...

cheers, MuS

SplunkTrust
SplunkTrust

please mark this as answered by ticking the tick - thx 😉

0 Karma

SplunkTrust
SplunkTrust

according to the docs http://docs.splunk.com/Documentation/Splunk/6.0.3/Troubleshooting/WhatSplunklogsaboutitself license_usage.log contains all information, not only top 10.
limits.conf is the place for the change, like I told you. Did you restart Splunk afterwards? Also this will only be valid for new events.

0 Karma

Explorer

Thank you for your answer MuS.

If we use the index=internal source=*licenseusage.log type=Usage we also get the top 10 sources and not all sources.

We have not found the value allowing to log all sources in *metrics.log and had to set arbitrarily the parameter in limits.conf
[metrics]
maxseries = 100000

Any help ?

Thanks in advance

0 Karma