Alerting

How to create live alert result into report to fetch historic data

akashsawant
Explorer

Present scenario:  We have alert " high memory "  detect systems if memory hits the set threshold ( if Committed Memory usage is over 115% ) - Running on schedule every hour at 15 minutes past hour 

Requirement :  This alert give us kind of live result but We need historic data in terms of report so that we can do review at the end of month to understand Host "Alpha" caught into this alert for x times in month , Host "beta" caught into this alert for y times ...so on. Basically need to find particular host how frequently and how many times having high memory.

 

| mstats avg(_value) AS CommittedMemoryInBytes WHERE index=xyz AND metric_name=Memory.Committed_Bytes  by host
| join host [search index=abc sourcetype=WHM source=operatingsystem TotalPhysicalMemoryKB=*]
| eval PercentCommittedMemory = round( (CommittedMemoryInBytes*pow(2,-30)) / (TotalPhysicalMemoryKB*pow(2,-20) )*100,2)
| where PercentCommittedMemory > 115
| table host,PercentCommittedMemory,CommittedMemoryInBytes,TotalPhysicalMemoryKB

 

 

Any help will be appreciated.

Labels (1)
0 Karma

venky1544
Builder

Hi @akashsawant 

you can either redirect the output to kvstore or standard lookups 

https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/uselookupswithkvstore/

and then retrieve the results for your historic data usecase 

 

if it helps karma is appreciated/if it resolves solution acceptance in appreciated

0 Karma

akashsawant
Explorer

Thanks @venky1544 

I created lookup 

akashsawant_0-1654090433757.png

but when I trying to throw in that lookup getting weird errors, DO you have any idea on this please ?

| mstats avg(_value) AS CommittedMemoryInBytes WHERE index=xyz AND metric_name=Memory.Committed_Bytes  by host
| join host [search index=abc sourcetype=WHM source=operatingsystem TotalPhysicalMemoryKB=*]
| eval PercentCommittedMemory = round( (CommittedMemoryInBytes*pow(2,-30)) / (TotalPhysicalMemoryKB*pow(2,-20) )*100,2)
| where PercentCommittedMemory > 115
| table host,PercentCommittedMemory,CommittedMemoryInBytes,TotalPhysicalMemoryKB
| outputlookup Akash-KV

akashsawant_1-1654090660851.png

 

 

0 Karma

akashsawant
Explorer

never mind @venky1544 , My bad i forgot to define KV store lookup.

Now I can able to throw my search result in Akash-KV.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...