How to set up email alerts on license warnings when I cannot access the license manager page?

[mataharry]

I am on an instance where I have no access to the license manager page, or where I never log in, or when I am not an admin (Splunk cloud or sandbox, in my case).

How can I set up a scheduled email alert to tell me when I exceed the license usage capacity?

[yannK · Accepted solution]

If you can search the internal logs, you can look for those events recorded just after midnight (on the server timezone, so for Splunk Cloud in GMT)

> 05-07-2015 00:00:00.047 +0000 WARN LicenseUsage - type=WarningIssued slave="C89XXXXX-32XX-46XX-95XXX-635XXXXXX99DE" stack="enterprise" pool="auto_generated_pool_enterprise" - This pool has exceeded its configured poolsize=XXXXXXXX bytes. A warning has been recorded for all members

you can setup a scheduled search running after midnight, and looking for the last 24h logs
index=_internal source=*license_usage.log* type=WarningIssued
and triggering if at least one events is returned, then email to you the result.

[rithwik572]

@yannK I would like to know how can I include the slave name (slave="C89XXXXX-32XX-46XX-95XXX-635XXXXXX99DE") in the alert generated ?

[yannK]

something like that can show you :
the last warning and the slave list.

  index=_internal source=*license_usage.log* type=WarningIssued | bucket _time=24h | stats count values(slave) first(_time) by time host

[rithwik572]

Hi @yannK,
I am trying to generate alert when a jenkins slave memory get's full. The search string I am using is index=app_devops AND "No space left on device" source="/opt/artifacts/jenkins/log/jenkins*"

Event's shown are as follows:
Caused by: java.io.IOException: remote file operation failed: /opt/jenkins/7cb92e15/tools/hudson.model.JDK/IBM_1.7_JDK at hudson.remoting.Channel@1901245e:Linux-build-slave-12: java.io.IOException: No space left on device

I would like to print the slave name ( Linux-build-slave-12) on the email body of alert generated in real time as we have 30 slaves running in our environment . current alert doesn't show the slave name, we need to look for full output to determine the slave name. Please let me know how to configure it,
Thanks in advance