So I'm confused.
Splunk has an option to set the maximum daily volume for a pool, yet it "SEEMS" that it will cause a violation.
I have a 50GB license, I know that sometimes folks may go over and I'd rather they didn't and since I don't go over all the time, I'm not ready to get a new license.
So say I set "Maximum daily volume this pool may consume" to 47GB. Now I "assume" that when I've indexed 47GB it will stop indexing, give me a nice warning and move on. I will obviously start indexing again at midnight, no big deal.
Now, at 47GB I get a warning, fine, that makes sense, I hit a configured amount and I should be told for my information, makes perfect sense. What doesn't make sense is Splunk treats this (apparently) as a warning to it's violation count, but I've not used my 50GB for the day, I've put in a speed bump to slow me down, stop me until midnight (okay I put in a red light not a speed bump).. So apparently what it seems is, I hit the warning 5 times in a given 2 week period and my splunk shutdown. But wait, I didn't use my 50GB a day, I only used 47GB, which means I had 3GB a day in reserve, so why does it appear that Splunk decided that my warnings should turn into a violation and I should not be able to use the software that I payed/paying for? This is a setting I put in, to control my behavior, i've not violated any usage terms and it seems like quite the opposite, Splunk has violated it's terms since I've not actually hit my 50GB cap, since I'm not allowing my system to use it, so I can't go over!
Is this Splunk being overzealous or can you explain how I'm misinterpreting this configuration param?
Splunk will not stop indexing when you reach your license limit. Are you sure you only ingested 47GB for that day, and not over 50GB? Check the License Usage Report for that day and see what you hit. Be sure to select All Pools, and not just the 47GB pool you configured.
Totally absolutely 100% sure! I got 5 days of warnings hitting 47GB and the 6th day I was locked out.
I only use one pool that contains my entire license (350GB), so I'm not terribly well-versed on license pools... with that said...
I believe the idea with pools is to be able to split your total license among groups at your organization. For example, if you configured your 50GB license as:
Total allocated: 50GB/50GB
You then point those groups to their own indexers (which are in their respective license pools). When that a certain group violates their license 5 times, they can no longer search their data -- but the other groups are unaffected. This is with the idea that this particular group requires more licensing, so you can juggle pools, or have them procure more licensing from Splunk for their needs.
If you have added all of your indexers to one pool, and this pool has 47GB of your 50GB license, leaving you with 3GB unallocated, each time you boink past 47GB, you will get a violation. Since your entire environment is essentially under one pool, you'll get locked out after 5 violations. You're essentially telling Splunk, "My license is only 47GB". Splunk won't stop indexing at 47GB (causing violations), but will violate you at any ingest between 47GB-50GB (and above, obviously).
If you wanted to "cap" Splunk indexing at your license volume and prevent possible violations, you either have to take a closer look at what you're indexing by blacklisting or whitelisting data, or create an alert that when you hit something like 95% of your license limit, triggers a script to disable indexing until midnight.
Also this is not my license limit, this is a "Maximum daily volume this pool may consume" setting. Which should absolutely be a hard stop. I know what you are saying about typical license, it will eat whatever you throw at it, so Splunk can make you call for a reset license 🙂