Perpetual 100GB license "exceeded" and locked out

[mgjk]

We're on a perpetual 100G license without support.  Everything is fine as we're not using it in mission-critical operations and up until a week ago, didn't have a high volume of logs.

Yesterday my reports failed because: "Error in 'litsearch' command: Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license "

There was no warning until the lockout.  Then the little bell was lit up in the dashboard.

What happens if I exceed my purchased license capacity?
The software will generate a warning if you exceed your licensed indexing volume on any one calendar day.

  - If you exceed your daily indexing limit five or more times in a rolling 30-day period and have a license stack volume over or equal to 100 GB on Splunk Enterprise version 6.5 and above, the software will not deactivate your search functionality.

Splunk Enterprise License Enforcement FAQ | Splunk .

Investigating our environment, a new data source was crazy, and exceeded 100GB/day for the last 5 days.  The crazy source was stopped, and now we're well below the threshold.  However we have 5 violations in the last 30 days, and it's going to stay that way for the next... 24 days or so.

We waited a day and see that we're still locked out.

The thought of paying for Splunk support for our perpetual license because we were locked out would not sit well with senior management. 

From what I read in the Splunk docs (above), a perpetual license 100GB+ is not supposed to lock out like this.  Our options seem to be:

1. Reinstall
2. Wait
3. Pay to tell support that the license enforcement went crazy and we need them to unbreak it

Are there any other tricks?

----

Update:  A colleague was able to get somebody at Splunk to correct the situation.  

Thanks for everyone's input.  We're putting some additional monitoring in place.

[PrewinThomas]

@mgjk 

Perpetual license ≥100GB/day should not disable your search. What does your license stack look like on the license server? If it's showing a trial license or a stack <100GB, that could trigger search disablement. If it's a proper 100GB license, try reinstalling the license and verify.
Otherwise, contact Splunk or your account manager to request a license reset key.

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[PickleRick]

Technically, as far as I remember, it's the license xml itself which tells Splunk whether the license is enforcing or not. Perpetual licenses are a bit unusual and are not covered by the normal docs so they might behave differently than your normal time-bound ones. I'm not sure Splunk issues perpetuals anymore so the license might have been issued back when there were different rules.

[mgjk · Accepted solution]

We have some good news, one of my colleagues found somebody at Splunk to speak to and they were able to get us up and running again.

Thanks for the thoughts.

[mgjk]

We do have a 100GB Splunk Enterprise version, but the support statement is wrong.  We're locked out of search. 

Our operations team reached out to Splunk support, they suggest we use an enterprise trial version or purchase a new Splunk license.

Apparently there's a difference between an "Active License" and a "Perpetual License", the later being inactive, although it's valid, which means it's legally active, but there's no support, even if the application breaks due to a bug in license enforcement.

I'm going to suggest the team forward to ELK, as it seems the heavy-forwarder capability is not impacted.

[PickleRick]

Technically, if you have a perpetual license without support, noone is obligated to do anything about your environment. If it works, it works. If it breaks, it breaks. You are allowed to use the software as per the license terms but you're not entitled to anything beyond that.

Having said that, you might call out to your local Splunk sales representative for help but results may vary.

[VatsalJagani]

This could be related to perpetual license, it might not follow the rules defined in the Doc, which I guess is for a regular Splunk license.

But you can still reach out to Splunk support and ask for a reset key.

[livehybrid]

Hi @mgjk 

Those docs state:

If you exceed your daily indexing limit five or more times in a rolling 30-day period and have a license stack volume over or equal to 100 GB on Splunk Enterprise version 6.5 and above, the software will not deactivate your search functionality.

Now that the installation is locked it can only be unrestricted using a reset key provided by Splunk which would usually be obtained by support. Are you able to raise any cases at all in the Support portal or does it not let you? 

If not you could try calling them to speak direct (see https://www.splunk.com/en_us/about-splunk/contact-us.html#customer-support) or reaching out to the sales team who may be able to help.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing