How do I shut off indexing for a certain group who are forwarding their data to my indexer when their data reaches a certain volume?

[fd26645]

Another team has asked me if they can send their syslog data to my Splunk server if they purchase some license capacity.

I know I can put limits on the amount of disk space they use through the index settings, but is there any way I can ensure that they do not exceed their license capacity and cause problems for me without me having check on things regularly?

Ideally I would like to create a separate UDP input that feeds to a separate index and if they exceed a the limit the Input should just "shut down" (just for that day if possible).

[stephane_cyrill · Accepted solution]

Hi,
to avoid that the license violation of an other team affect you:

1- you can divide your license stack into many license pool. and by doing so you create a specific pool for that team with a limit license quota, Index violations are then contained within the specific pool.
For example if that team have a license capacity of 4GB you assign it to a pool of 4GB too.
So to minimize the damage you should have as many "pools" as you have license slaves; one slave per pool or do as you like.

2- For that team you can create an alert that send an sms/email to the admin of that team went the license violation is near(ei when they reach 85% of thier total daily license volume) so that he can stop.

[dominiquevocat]

hm, that sure sounds nice. I tried something along the lines i.e. i have a pool that is smaller then the total license and assigned it to a indexer. However if the forwarders send more data to that indexer then allowed by this indexers' pool size it will continue and kill the total license.

example:
Total license 50GB
Integration indexer: 1GB
Production Indexer: 48GB

Result is it still occasionally kills the license. Is there something i make wrong? What else can i set to prevent outrunning the total license?

[rsennett_splunk]

Just to be clear, Stephane's answer refers to indexERS or, more clearly... separate instances of Splunk which act as slaves to the master, and point to different pools on the master. You cannot do that with individual indexes which is what the question asks...

[stephane_cyrill]

you are right rsennet_splunk.

[rsennett_splunk]

Theoretically, you could monitor (set an alert) their sourcetype or better the index you create for them, and kick off a script that disables the input. But that's not really helpful to anyone as I'm sure the data they are forwarding is of some value to them, and you really can't know what it is that will not be indexed because of that.

If the data is coming in via forwarder, and you are using the Deployment Server to manage the forwarders... your script would edit the inputs.conf in the $SPLUNK_HOME/etc/deployment-apps/ . The next time the config was checked for changes, the new config with "disabled=true" would be sent to the forwarder and it would stop sending the data...

A better plan, would be to have them purchase enough license to accommodate their data, and keep an eye on it via the licensing dashboard. (you can use those searches to create alerts if you like also) You can see which source, sourcetype, or index is using what amount of volume.

The actual license monitoring is pretty friendly... The first four times that you violate the Splunk license, you'll see a message display telling you the volume. The fifth time in a 30 day (rolling) period that the license is violated, Search is disabled, and you'll need to contact support (or your Account Team) to send you a reset key. In other words, if they blow out your license, or they eat too much of your license... you do have a chance to cut them off. It's usually a good idea to actually let it run and see what the pattern is. I don't know what the data is that they're sending, but it would really stink if something went down after you've cut them off and the data is not there to investigate...

The only way to make their indexing volume completely independent of yours, is to give them their own indexer, make it a license slave and attach it to a license pool that you've carved off of your own license. The individual pools are completely independent of each other. When one violates enough to shut off search, the indexing volume for that pool is not counted towards anything at that point... until you reset the pool (the 30 day clock). That's not going to help you with them blowing up your disk if all the indexers have $SPLUNK_DB/ pointing to the same place, but you won't blow out your license because they've violated their slice.

[gehogan3]

I am running into this same problem...our QA group requested a portion of our production splunk license pool...so I set up their own indexer and set it up to use a pool carved out from the production volume license. The problem is that if they have a runaway server that spews a ton of log data during automated testing overnight, it CAN affect our production license pool. The QA data blows through their pool license and gets the nice little warning, but the data KEEPS GETTING INDEXED...which means it's now chewing up part of my production license....and if left unchecked can violate that license.

Without manual intervention, how can I get the QA indexer to stop indexing the traffic from the offending forwarder?

[rsennett_splunk]

Nope. it keeps getting indexed, but it doesn't count towards anything. Each pool is independent... if left unchecked... it's just unchecked (and you can't search as you will have violated the pool quota) but it has no affect on any other pool.

[fd26645]

I feel like the edit to my question title is not accurate. I will be managing all of the licensing. I just want to make sure that this team does not exceed the license capacity I purchased for them and begin eating into the license capacity I purchased for my own environment. Maybe this detail is insignificant.

[rsennett_splunk]

I agree... so I've edited it again...

[fd26645]

I like this title better thanks.