Splunk License Usage Alerting

[anandhalagaras1]

Hi Team,

We have opted for 250 GB of licensing on daily basis.  So if the license is reaching more than 70% (i.e. 175 GB) i need to get an alert similarly if the license is getting reached 80% and more (i.e. 200 GB) then i need to get another alert. And finally if it crossed more than 90% (i.e. 225 GB) i need to get another alert.

So can you help me with the Search query.

[Mitesh_Gajjar]

Hi @anandhalagaras1 

You can try this query:

| rest /services/licenser/pools
| eval total_quota_gb = toint(usage_quota / 1024 / 1024 / 1024)
| eval used_gb = toint(usage_used / 1024 / 1024 / 1024)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table total_quota_gb, used_gb, usage_percentage
| where usage_percentage >= 70 AND usage_percentage < 80
| eval alert_level = "70%-79%"
| eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action."
| if(usage_percentage >= 80 AND usage_percentage < 90, appendpipe [| eval alert_level = "80%-89%"; eval alert_message = "License usage has reached " . usage_percentage . "%. Please take immediate action."], "")
| if(usage_percentage >= 90, appendpipe [| eval alert_level = "90% and above"; eval alert_message = "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!"], "")
| table alert_level, alert_message

[anandhalagaras1]

@Mitesh_Gajjar ,

When i use the search query i am getting an error as below:

Unknown search command 'if'.

So kindly help to check and update on the same.

[Mitesh_Gajjar]

Hi @anandhalagaras1, you can try this query. 

rest /services/licenser/pools
| eval total_quota_gb = round(usage_quota / (1024 * 1024 * 1024), 2)
| eval used_gb = round(usage_used / (1024 * 1024 * 1024), 2)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table total_quota_gb, used_gb, usage_percentage
| where usage_percentage >= 70 AND usage_percentage < 80
| eval alert_level = "70%-79%"
| eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action."
| eval alert_level = if(usage_percentage >= 80 AND usage_percentage < 90, "80%-89%", alert_level)
| eval alert_message = if(usage_percentage >= 80 AND usage_percentage < 90, "License usage has reached " . usage_percentage . "%. Please take immediate action.", alert_message)
| eval alert_level = if(usage_percentage >= 90, "90% and above", alert_level)
| eval alert_message = if(usage_percentage >= 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!", alert_message)
| table alert_level, alert_message

[anandhalagaras1]

@Mitesh_Gajjar I am not getting any results eventhough i ran the search query for All Time. We are using Splunk Cloud and the Cloud Monitoring Console app is installed in our Search Head.

So I have tried the query in Search and Reporting App of the SH and also CMC app Search but no results. But actually within last 60 days we had multiple breaches occurred in over all licensing. 

[gcusello]

Hi @anandhalagaras1 ,

there's an error, the eval command is missed before the if, anyway, please, try to use the search in the Monitoring Console:

| rest splunk_server_group=dmc_group_license_master /services/licenser/pools 
| join type=outer stack_id splunk_server [rest splunk_server_group=dmc_group_license_master /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields splunk_server stack_id is_active] 
| search is_active=1 
| fields splunk_server, stack_id, used_bytes 
| join type=outer stack_id splunk_server [rest splunk_server_group=dmc_group_license_master /services/licenser/stacks | eval stack_id=title | eval stack_quota=quota | fields splunk_server stack_id stack_quota] 
| stats sum(used_bytes) as used_bytes max(stack_quota) as stack_quota by splunk_server 
| eval usedGB=round(used_bytes/1024/1024/1024,3) 
| eval totalGB=round(stack_quota/1024/1024/1024,3) 
| eval percentage=round(usedGB / totalGB, 3)*100 
| fields splunk_server, percentage, usedGB, totalGB 
| where percentage > 80 
| rename splunk_server AS Instance, percentage AS "License quota used (%)", usedGB AS "License quota used (GB)", totalGB as "Total license quota (GB)"

Ciao.

Giuseppe

[anandhalagaras1]

 When I try to use the query I am not getting any results. I tried in Search and Reporting app as well as in CMC app.

[gcusello]

Hi @anandhalagaras1 ,

where did you run this search? you should try on the License Master.

Ciao,

Giuseppe

[anandhalagaras1]

@gcusello Our Splunk is hosted in Cloud and it is managed by Splunk Support.

We have access only to Search heads and not to License master server.

[Mitesh_Gajjar]

you can run below query in your CMC.

| rest splunk_server_group=* /services/licenser/pools
| eval total_quota_gb = round(your_quota_field / (1024 * 1024 * 1024), 2)
| eval used_gb = round(your_used_field / (1024 * 1024 * 1024), 2)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table splunk_server, total_quota_gb, used_gb, usage_percentage
| eval alert_level = case(
usage_percentage > 90, "Critical",
usage_percentage >= 80, "High",
usage_percentage >= 70, "Medium",
true(), "Normal"
)
| eval alert_message = case(
usage_percentage > 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!",
usage_percentage >= 80, "License usage has reached " . usage_percentage . "%. Please take immediate action.",
usage_percentage >= 70, "License usage has reached " . usage_percentage . "%. Please take action.",
true(), "License usage is within normal range."
)
| where usage_percentage > 70
| table splunk_server, total_quota_gb, used_gb, usage_percentage, alert_level, alert_message

Make sure to replace your_quota_field & your_used_field with the correct field name representing the license quota in your Splunk Cloud environment.

[gcusello]

Hi @anandhalagaras1 ,

in this case see in the Cloud Monitoring Console App at https://<your_instance>.splunkcloud.com/en-US/app/splunk_instance_monitoring/alerts, you can find the aler named "CMC Alert - Ingest Volume Exceeds 80%".

You can open in search this alert and enable it:

the search is 

`sim_licensing_summary_base` 
      | `sim_licensing_summary_no_split` 
      | append 
          [| search `sim_licensing_limit`] 
      | stats latest(GB) as usage latest("license limit") as limit 
      | eval ratio = usage/limit 
      | where ratio > .8

but maybe the macros don't run outside this app, but you can run it in the app.

if you want to use it outside the app, you should replae the macros.

Ciao.

Giuseppe

[gcusello]

Hi @anandhalagaras1,

in the Monitoring Console there's the alert you require, it's named: "DMC Alert - Total License Usage Near Daily Quota".

you can find it at http://your_splunk_server:8000/en-US/app/splunk_monitoring_console/alerts

Ciao.

Giuseppe