How to create live alert result into report to fetch historic data

[akashsawant]

Present scenario:  We have alert " high memory "  detect systems if memory hits the set threshold ( if Committed Memory usage is over 115% ) - Running on schedule every hour at 15 minutes past hour 

Requirement :  This alert give us kind of live result but We need historic data in terms of report so that we can do review at the end of month to understand Host "Alpha" caught into this alert for x times in month , Host "beta" caught into this alert for y times ...so on. Basically need to find particular host how frequently and how many times having high memory.

| mstats avg(_value) AS CommittedMemoryInBytes WHERE index=xyz AND metric_name=Memory.Committed_Bytes  by host
| join host [search index=abc sourcetype=WHM source=operatingsystem TotalPhysicalMemoryKB=*]
| eval PercentCommittedMemory = round( (CommittedMemoryInBytes*pow(2,-30)) / (TotalPhysicalMemoryKB*pow(2,-20) )*100,2)
| where PercentCommittedMemory > 115
| table host,PercentCommittedMemory,CommittedMemoryInBytes,TotalPhysicalMemoryKB

Any help will be appreciated.

[venky1544]

Hi @akashsawant 

you can either redirect the output to kvstore or standard lookups 

https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/uselookupswithkvstore/

and then retrieve the results for your historic data usecase 

if it helps karma is appreciated/if it resolves solution acceptance in appreciated

[akashsawant]

Thanks @venky1544 

I created lookup 

but when I trying to throw in that lookup getting weird errors, DO you have any idea on this please ?

| mstats avg(_value) AS CommittedMemoryInBytes WHERE index=xyz AND metric_name=Memory.Committed_Bytes  by host
| join host [search index=abc sourcetype=WHM source=operatingsystem TotalPhysicalMemoryKB=*]
| eval PercentCommittedMemory = round( (CommittedMemoryInBytes*pow(2,-30)) / (TotalPhysicalMemoryKB*pow(2,-20) )*100,2)
| where PercentCommittedMemory > 115
| table host,PercentCommittedMemory,CommittedMemoryInBytes,TotalPhysicalMemoryKB
| outputlookup Akash-KV

[akashsawant]

never mind @venky1544 , My bad i forgot to define KV store lookup.

Now I can able to throw my search result in Akash-KV.