Archive
Highlighted

Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Motivator

Hi All, Currently we could License warning message popping out in splunk web portal ? This is the firs warning message we have got for this month and as per splunk document we are allowed for five violation in 30 day period. Similarly when checked in the license manager under Enterprise license group --> alerts -- > Current & Permanent

Current : 1 pool warning reported by 1 indexer
Permanent : 10 license window warnings reported by 8 indexers

Volume used today is under the threshold limit, but still we are getting this warning message
Effective daily volume is 348 GB (License limit)
autogeneratedpool_enterprise is 187 GB

When checked Usage report Today's License Usage (GB) we could see the license usage is increasing constantly.

Kindly let me know how to troubleshoot this issue and control the license before it goes out of hands.
And also let me know how to figure out which source/source type/host are consuming more licenses.

Details : Splunk 6.2.1 version
License type : Enterprise License.

thanks in advance

Tags (2)
0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

SplunkTrust
SplunkTrust

in youre license master, navigate to settings (top tight corner) -> licensing - > usage report
if you click last 30 days, you can split by host, index, source and sourcetype.
here is a good start for searches and reports about your license: http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Motivator

thanks adonio, for sharing the link.

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

SplunkTrust
SplunkTrust

Perhaps you can read About the Splunk Enterprise license usage report view the report provides the ability to breakdown the data by sourcetype/source et cetera.
Alternatively there are many queries on this forum to do the same thing by viewing the license data in the _internal index.

Once you determine what is causing the issue you then need to either buy more license or decrease the amount ingested by turning off unnecessary sources, or filter your incoming data using transforms to drop the data you do not want (eg. you can drop each line of a log file containing the keyword DEBUG or similar).

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Champion

/// This is the firs warning message we have got for this month and as per splunk document we are allowed for five violation in 30 day period.
///

Yes, Splunk allows 5 violations within a month and this is your first violation.
No issues, This is a normal scenario.

I think you thought/assumed like - "ONLY after 5 violations, we will get warnings".
But it is not like that.
After 5 violations, the search functionality will not be available.

Currently I hope you have search functionality working fine.

And also let me know how to figure out which source/source type/host are consuming more licenses

You can find troubleshooting steps at -
http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Motivator

thanks for sharing the link.

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Champion

Hi Hemnaath, do you have any other questions related to this issue?, if no, can you please mark this question as answered, accept as answer, please.

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Motivator

Hi Ventsekar, today also we have got another warning for licenses and when checked in the usage report , i am unable to identify which source, index, source type is consuming more license in our environment, can you share me the exact query to find out which index,source and sourcetype is consuming more licenses in GB for past 24 hrs.

thanks in advance.

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

SplunkTrust
SplunkTrust

index=internal host= source=*licenseusage.log type=Usage | eval MB=round((b/1024)/1024,2) | bin _time span=1d |stats sum(MB) AS usage(mb) by h, _time | addcoltotals

That will divide by host, h is host, idx is the index, s is source et cetera, just look through the raw data in the:
index=internal host= source=*licenseusage.log type=Usage

If you need more information on this 🙂

0 Karma
Highlighted

Re: Daily indexing volume limit exceeded for 8 slaves ? Today this message is popping out in splunk web.

Motivator

Hi garethatiag, thanks for your query and I had modified ur query to find out which source type and index is eating the licenses more on a day.

source=license_usage.log type="Usage" idx=*
| stats sum(gb) as Totalcount by st,idx | sort - Totalcount | eventstats sum(Totalcount) as SUM | eval P=round(((Totalcount/SUM)*100),2)|eval Percentage=P+" "+"%" | table st idx Totalcount SUM Percentage.

thanks.

0 Karma