Archive

Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Hemnaath
Motivator

Hi All, Currently we could License warning message popping out in splunk web portal ? This is the firs warning message we have got for this month and as per splunk document we are allowed for five violation in 30 day period. Similarly when checked in the license manager under Enterprise license group --> alerts -- > Current & Permanent

Current : 1 pool warning reported by 1 indexer
Permanent : 10 license window warnings reported by 8 indexers

Volume used today is under the threshold limit, but still we are getting this warning message
Effective daily volume is 348 GB (License limit)
auto_generated_pool_enterprise is 187 GB

When checked Usage report Today's License Usage (GB) we could see the license usage is increasing constantly.

Kindly let me know how to troubleshoot this issue and control the license before it goes out of hands.
And also let me know how to figure out which source/source type/host are consuming more licenses.

Details : Splunk 6.2.1 version
License type : Enterprise License.

thanks in advance

Tags (2)
0 Karma

Hemnaath
Motivator

Hi garethatiag, thanks for your query and I had modified ur query to find out which source type and index is eating the licenses more on a day.

source=license_usage.log type="Usage" idx=*
| stats sum(gb) as Totalcount by st,idx | sort - Totalcount | eventstats sum(Totalcount) as SUM | eval P=round(((Totalcount/SUM)*100),2)|eval Percentage=P+" "+"%" | table st idx Totalcount SUM Percentage.

thanks.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Not a problem, it might be a good idea to accept one of the answers so everyone knows your question is answered...and/or upvote where appropriate 🙂

0 Karma

Hemnaath
Motivator

Hi garethatiag, we had got another Waring within this week and after executing the above query, we got to know that firewall and windows data are consuming more then 50% of the licenses and there are some abnormal spike at early hours, so wanted to investigate on why there was an spike. Could guide me how to approach to investigate.

thanks in advance.

0 Karma

gjanders
SplunkTrust
SplunkTrust

I can only provide general ideas, you can see from the license report what times the data was most used.
I would consider starting with something simple like:
| tstats count where index= groupby _time span=1h

Or similar and looking for the point where large numbers of events come in, and then just looking at what kind of event, did (ie. index= , perhaps with sampling turned on) and either use the patterns button or look for a particular source or a pattern in the data...
Good luck

0 Karma

Hemnaath
Motivator

thanks garethatiag, we found that during midnight there is a huge spike and based on the license report, we found the source estreamer from a particular host was consuming more licenses and in turn when checked with management these data were important for the security team. So we request to purchase additional amount of license.

0 Karma

inventsekar
Super Champion

/// This is the firs warning message we have got for this month and as per splunk document we are allowed for five violation in 30 day period.
///

Yes, Splunk allows 5 violations within a month and this is your first violation.
No issues, This is a normal scenario.

I think you thought/assumed like - "ONLY after 5 violations, we will get warnings".
But it is not like that.
After 5 violations, the search functionality will not be available.

Currently I hope you have search functionality working fine.

And also let me know how to figure out which source/source type/host are consuming more licenses

You can find troubleshooting steps at -
http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

0 Karma

Hemnaath
Motivator

thanks for sharing the link.

0 Karma

inventsekar
Super Champion

Hi Hemnaath, do you have any other questions related to this issue?, if no, can you please mark this question as answered, accept as answer, please.

0 Karma

Hemnaath
Motivator

Hi Ventsekar, today also we have got another warning for licenses and when checked in the usage report , i am unable to identify which source, index, source type is consuming more license in our environment, can you share me the exact query to find out which index,source and sourcetype is consuming more licenses in GB for past 24 hrs.

thanks in advance.

0 Karma

gjanders
SplunkTrust
SplunkTrust

index=_internal host= source=*license_usage.log type=Usage | eval MB=round((b/1024)/1024,2) | bin _time span=1d |stats sum(MB) AS usage(mb) by h, _time | addcoltotals

That will divide by host, h is host, idx is the index, s is source et cetera, just look through the raw data in the:
index=_internal host= source=*license_usage.log type=Usage

If you need more information on this 🙂

0 Karma

gjanders
SplunkTrust
SplunkTrust

Perhaps you can read About the Splunk Enterprise license usage report view the report provides the ability to breakdown the data by sourcetype/source et cetera.
Alternatively there are many queries on this forum to do the same thing by viewing the license data in the _internal index.

Once you determine what is causing the issue you then need to either buy more license or decrease the amount ingested by turning off unnecessary sources, or filter your incoming data using transforms to drop the data you do not want (eg. you can drop each line of a log file containing the keyword DEBUG or similar).

0 Karma

adonio
SplunkTrust
SplunkTrust

in youre license master, navigate to settings (top tight corner) -> licensing - > usage report
if you click last 30 days, you can split by host, index, source and sourcetype.
here is a good start for searches and reports about your license: http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

0 Karma

Hemnaath
Motivator

thanks adonio, for sharing the link.

0 Karma