license manager results differ from metrics.log indexed volume

[att35]

Hi,

This morning I saw message regarding daily indexing volume being exceeded. From the license manager, it seems that one of the indexers processed around 8 GB(We have 4 Indexers, with master/slave configuration).

I ran the following search to find out which source indexed the most, to figure out the sudden spike.

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart sum(kb) by series | sort - sum(kb)

But when I add the individual volume, it comes out around 4 GB, half of what license manager reports. Am I looking at a incorrect source(metrics.log) for this data? This search was executed directly on the indexer which show high index vol, whereas license manager is on the search head(which also acts as license master).

If we assume license manager to be the correct result, how can I further drill down to find the source or host that sent most data.

Many Thanks,

~ Abhi

[masonmorales · Accepted solution]

There's a pretty good app for diagnosing license utilization available on Splunkbase: Splunk Utilization Monitor (SUM)

[att35]

Hi Mason,

First of all, Thank you writing a nice all-in-one app for licensing.. 🙂

In a master-slave setup, where should we install this? I believe it should be on license master.. correct?

I installed it on both, the license master as well as one of the search peers(license slave), but it is unable to populate any data. The drop down for Splunk server & pool display "Search produced no results".. whereas the panels are either "no results found" or "NA".

Is there any manual configuration that I might be missing?

Thanks,

~Abhi

[masonmorales]

Hi Abhi,

That's correct, you should install the app on your license master. The drop-downs are populated using the splunk_server and pool fields from:

index=_internal source=*license_usage.log

Please ensure that you are able to search index=_internal on the user account that you are using the App with. If you cannot, you will need to login with a user that has higher privileges (e.g. admin), or go to Settings -> Access controls -> Roles -> (Your Role) and ensure that the _internal index is listed under the "selected search indexes"

[att35]

Hi Mason,

That was the issue. The role used to view the license usage did not have permissions on the internal indexers. It's working perfectly now.

Also, I did notice that there's some difference in the indexed volume reported by SUM as compared with the default "License Usage" app.. In the default app, today's usage is 2.54 GB, but within SUM, the first panel for "License Pool Utilization (GB) (Today)" reports 4.41 GB.

I'll do some more checking on my side to see if there's something in the query.

Thanks again for the help,

~Abhi

[yannK]

you should ask the app's author -> https://splunkbase.splunk.com/app/2678/

or open the searches and look at them, maybe the host is hardcoded in it.

[yannK]

Remember that the metrics.log contains a sample of top 10 only
http://docs.splunk.com/Documentation/Splunk/6.2.2/Troubleshooting/Aboutmetricslog#Thruput_messages

for any license usage, You should trust the searches based on license_usage.log

[masonmorales]

Thanks for pointing this out. I've updated two of my panels that were using metrics.log in the SUM app.

[stephane_cyrill]

Hi ,
1- The sum of the indexed volume may have been above the limit of your licenses.

2- assuming that the license manager result is correct, to find the soure or host that sent most data you :
find the max(kb) by host or by source over that period of time.

3- for example:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart max(kb) by host

[nivedita_viswan]

I have the same issue!

We have a 20GB license.
The sum of the indexed volume for the last day is just 6GB! I still see a violation for yesterday. Here is the search query:

index="_internal" group="per_index_thruput" | search series!="_audit" | search series!="_internal"| search series!="_introspection"| eval gb=kb/1024/1024 | timechart span="1d" sum(gb) by series

Furthermore, this was the 5th violation so my search has been suspended though the indexed volume was just 6GB the last day.