If I have 2 x 100GB license files, creating a 200GB stack.
Then this stack, on a single licensing master is split into 3 licensing pools and is then allocated against individual 3 indexers.
Pool 1 = 50GB, allocated to indexer 1
Pool 2 = 50GB, allocated to indexer 2
Pool 3 = 100GB, allocated to indexer 3
Is the violation scoped to the pool or the stack?
eg. Am I right in saying that if the total of violations across Pool 1, Pool 2 and Pool 3 exceed 5 that searching across indexer 1, indexer 2 and indexer 3 will be disabled? Or is the violation just scoped to the violating pool, hence a single indexer? eg. Pool 1 is in violation only cause indexer 1 to have searching disabled.
I assume the entire stack is violated; How can this impact be limited?
Is there a suitable strategy to avoid a rogue (occasional high volume) indexer adversely affecting other indexers?
Is dedicated license files per Indexer a (possible expensive) solution?
When a Licensing 'alert' occurs can the pool allocations be juggled around prior to midnight to avoid a licensing warning? Is this the standard strategy?
It is entirely possible that a pool will occasionally violate its license - for example, if your infrastructure is having a really bad day. That's one reason that Splunk licensing is set up as it is: on that day (when your infrastructure is crashing around you), you really need Splunk, regardless of the consequences to your license. Even if your total license is violated on a single day, Splunk will continue to run without any consequences.
Remember that you get 5 violations (for an enterprise license) before search is locked - so don't panic, just monitor and plan.
And yes, you can "juggle" the pool allocations as needed before midnight to avoid a warning. As long as the total license is not exceeded, this can be a viable strategy. It's really a matter of how you want to allocate your licenses for your company's use of Splunk and how much monitoring/juggling you want to do at the pool level.
You can certainly assign a separate license to each indexer, but that can be an expensive and hard-to-manage solution. Most people just put all their licenses in a single pool (the default). That way, violations occur only if the total license is violated 5 times - individual indexers may be more or less busy, but it may not cause the aggregate to exceed the license.