splunk 6.1 error and cannot search : This pool has exceeded its configured poolsize

[jgauruder1]

 

splunk 6.1 error and cannot search :

Error in 'litsearch' command: Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting www.splunk.com....

 

The search job has failed due to an error. You may be able view the job in the Job Inspector

when i check settings->system->licensing and click "show all messages, there are 5 messages on

Nov 3rd, 4th, 7th, 8th, 9th

"This pool has exceeded its configured poolsize=21474836480 bytes. A warning has been recorded for all members"

  How do we tshoot and resolve this to get search working again?

We do not have an active splunk support contract.

Regards,

Jason

[isoutamo]

Hi

this error means that you have hard license violation and you cannot run regular searches before you get reset license or your daily ingest is less than your license amount enough long time so you get rid of licensing violation.

As you haven’t valid support contract I don’t believe that you get reset license. So your options are try to buy support contract or just wait until your license violation has resolved by indexing less (max 4 violations by 30 days, if I recall right for this version?).

Anyhow your version is quite old (dropped out from support already several years ago), that you should update it if it’s still in use.

r. Ismo

[PickleRick]

I don't know how it worked back in 6.1 but in "modern" versions if you're out of license (it expired), it's treated as if you had violations and your search is blocked. You can't just reset it. You need active license to keep your Splunk searchable. And you need the reset license to unlock it.

[jgauruder1]

we ended up doing a full system restore from backup to the days prior to the start of the warning messages in splunk.

so now search works without error and licensing shows normal,  and as expected, we lose data from the days after backup to the point of restore.  so for example, if I try to search for "yesterday" i get no results.  but that is the price paid for restoring from backup.

I guess the question that remains is : how can we in the future "see" what syslog client (or clients) is causing a license warning to be triggered ?  perhaps some security appliance sent an extended (many hours or more) burst of syslogs above the normal rate...but is there an easy way to see that in the splunk web ui ?

Regards,

jason

[isoutamo]

I’m not sure if there was a DMC or was this before it? If it was already published maybe there was Licensing views where you could try to see what sourc/host/sourcetype was cathode bursts? Another option was try to find SoS app which (maybe) could show this to you? And last option is try to look if this information has stored to _internal index? Worst case is that you must write your own report to check events’ lengths and calculate summaries based on that.