All Apps and Add-ons

Splunk License Usage Alerting

anandhalagaras1
Communicator

Hi Team,

We have opted for 250 GB of licensing on daily basis.  So if the license is reaching more than 70% (i.e. 175 GB) i need to get an alert similarly if the license is getting reached 80% and more (i.e. 200 GB) then i need to get another alert. And finally if it crossed more than 90% (i.e. 225 GB) i need to get another alert.

 

So can you help me with the Search query.

Labels (2)
0 Karma

Mitesh_Gajjar
Loves-to-Learn

Hi @anandhalagaras1 

You can try this query:

| rest /services/licenser/pools
| eval total_quota_gb = toint(usage_quota / 1024 / 1024 / 1024)
| eval used_gb = toint(usage_used / 1024 / 1024 / 1024)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table total_quota_gb, used_gb, usage_percentage
| where usage_percentage >= 70 AND usage_percentage < 80
| eval alert_level = "70%-79%"
| eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action."
| if(usage_percentage >= 80 AND usage_percentage < 90, appendpipe [| eval alert_level = "80%-89%"; eval alert_message = "License usage has reached " . usage_percentage . "%. Please take immediate action."], "")
| if(usage_percentage >= 90, appendpipe [| eval alert_level = "90% and above"; eval alert_message = "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!"], "")
| table alert_level, alert_message
0 Karma

anandhalagaras1
Communicator

@Mitesh_Gajjar ,

When i use the search query i am getting an error as below:

Unknown search command 'if'.

So kindly help to check and update on the same.

0 Karma

Mitesh_Gajjar
Loves-to-Learn

Hi @anandhalagaras1, you can try this query. 

rest /services/licenser/pools
| eval total_quota_gb = round(usage_quota / (1024 * 1024 * 1024), 2)
| eval used_gb = round(usage_used / (1024 * 1024 * 1024), 2)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table total_quota_gb, used_gb, usage_percentage
| where usage_percentage >= 70 AND usage_percentage < 80
| eval alert_level = "70%-79%"
| eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action."
| eval alert_level = if(usage_percentage >= 80 AND usage_percentage < 90, "80%-89%", alert_level)
| eval alert_message = if(usage_percentage >= 80 AND usage_percentage < 90, "License usage has reached " . usage_percentage . "%. Please take immediate action.", alert_message)
| eval alert_level = if(usage_percentage >= 90, "90% and above", alert_level)
| eval alert_message = if(usage_percentage >= 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!", alert_message)
| table alert_level, alert_message

0 Karma

anandhalagaras1
Communicator

@Mitesh_Gajjar I am not getting any results eventhough i ran the search query for All Time. We are using Splunk Cloud and the Cloud Monitoring Console app is installed in our Search Head.

So I have tried the query in Search and Reporting App of the SH and also CMC app Search but no results. But actually within last 60 days we had multiple breaches occurred in over all licensing. 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

there's an error, the eval command is missed before the if, anyway, please, try to use the search in the Monitoring Console:

| rest splunk_server_group=dmc_group_license_master /services/licenser/pools 
| join type=outer stack_id splunk_server [rest splunk_server_group=dmc_group_license_master /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields splunk_server stack_id is_active] 
| search is_active=1 
| fields splunk_server, stack_id, used_bytes 
| join type=outer stack_id splunk_server [rest splunk_server_group=dmc_group_license_master /services/licenser/stacks | eval stack_id=title | eval stack_quota=quota | fields splunk_server stack_id stack_quota] 
| stats sum(used_bytes) as used_bytes max(stack_quota) as stack_quota by splunk_server 
| eval usedGB=round(used_bytes/1024/1024/1024,3) 
| eval totalGB=round(stack_quota/1024/1024/1024,3) 
| eval percentage=round(usedGB / totalGB, 3)*100 
| fields splunk_server, percentage, usedGB, totalGB 
| where percentage > 80 
| rename splunk_server AS Instance, percentage AS "License quota used (%)", usedGB AS "License quota used (GB)", totalGB as "Total license quota (GB)"

Ciao.

Giuseppe

 

0 Karma

anandhalagaras1
Communicator

 When I try to use the query I am not getting any results. I tried in Search and Reporting app as well as in CMC app.

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

where did you run this search? you should try on the License Master.

Ciao,

Giuseppe

0 Karma

anandhalagaras1
Communicator

@gcusello Our Splunk is hosted in Cloud and it is managed by Splunk Support.

We have access only to Search heads and not to License master server.

0 Karma

Mitesh_Gajjar
Loves-to-Learn

you can run below query in your CMC.

| rest splunk_server_group=* /services/licenser/pools
| eval total_quota_gb = round(your_quota_field / (1024 * 1024 * 1024), 2)
| eval used_gb = round(your_used_field / (1024 * 1024 * 1024), 2)
| eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2)
| table splunk_server, total_quota_gb, used_gb, usage_percentage
| eval alert_level = case(
usage_percentage > 90, "Critical",
usage_percentage >= 80, "High",
usage_percentage >= 70, "Medium",
true(), "Normal"
)
| eval alert_message = case(
usage_percentage > 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!",
usage_percentage >= 80, "License usage has reached " . usage_percentage . "%. Please take immediate action.",
usage_percentage >= 70, "License usage has reached " . usage_percentage . "%. Please take action.",
true(), "License usage is within normal range."
)
| where usage_percentage > 70
| table splunk_server, total_quota_gb, used_gb, usage_percentage, alert_level, alert_message

 

Make sure to replace your_quota_field & your_used_field with the correct field name representing the license quota in your Splunk Cloud environment.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

in this case see in the Cloud Monitoring Console App at https://<your_instance>.splunkcloud.com/en-US/app/splunk_instance_monitoring/alerts, you can find the aler named "CMC Alert - Ingest Volume Exceeds 80%".

You can open in search this alert and enable it:

the search is 

`sim_licensing_summary_base` 
      | `sim_licensing_summary_no_split` 
      | append 
          [| search `sim_licensing_limit`] 
      | stats latest(GB) as usage latest("license limit") as limit 
      | eval ratio = usage/limit 
      | where ratio > .8

but maybe the macros don't run outside this app, but you can run it in the app.

if you want to use it outside the app, you should replae the macros.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

in the Monitoring Console there's the alert you require, it's named: "DMC Alert - Total License Usage Near Daily Quota".

you can find it at http://your_splunk_server:8000/en-US/app/splunk_monitoring_console/alerts

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...