I have this search
index=_internal source="*metrics*" group="per_index_thruput" series="customindex" host="*MyIndexers*"
this gives me the sum(kb) for each "customindex" that is recorded on each indexers metrics.log
When I do this
index=_internal source="*metrics*" group="per_index_thruput" series="customindex" host!="*MyIndexers*"
this gives me the sum(kb) for each "customindex" that is recorded on each hosts metrics.log
Since the host is sending and the indexers are receiving how is it that the hosts are sending less than what is being received?
Because the metrics.log shows a pooling of the top 10 values every 30 seconds.
Therefore , if you have more than 10 hosts, you will only see the first 10 in the metrics.
see the remarks here : http://docs.splunk.com/Documentation/Splunk/6.1.4/Troubleshooting/Aboutmetricslog#Thruput_messages
If you want more precise indexed volume, you can look at the
index=_internal source=*license_usage.log* type=Usage | stats sum(b) by idx
see http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume
Thanks @yannK Not yet on 6.1 I will be upgrading soon so I don't have the idx value in the license_usage.log
I'm looking through the 6.1.4 troubleshooting to see if this can help me with my 4.3 version until the upgrade planed in three weeks
Thanks for the help 🙂
@yannK
When I use license_usage.log I get half the amount of volume count but when I use source="*metrics.log"
I get the twice the amount of volume compared to that of license_usage.log
When I use
index="_internal" source=license_usage.log type=Usage | eval b=b/(1024*1024) |timechart span=d sum(b)
I get 49 GB for a specific Day
AND
When i use
index="_internal" source="*metrics.log" per_index_thruput | eval GB=kb/(1024*1024) |timechart span=d sum(GB)
I get 98GB for that same day.
So as I understand metrics.log will only return top 10 values every second and will not give precise data?
But seems to be otherwise.
You may be are comparing Apples and Oranges.
A side remark : when you look at logs, always add a * at the end of the source, in case the logs rotated, and some event are reported under the rotated log source. (source=license_usage.log* can find data license_usage.log.1 )
Thanks @ppablo_splunk for the edits to make the title more understandable. I appreciate your help.
No problem @hartfoml 🙂 I appreciate the thanks!