Alerting

How to compare a json value field in a 1h window, and detect a 50% increase in the value

matiasburni
Engager

Hi all,

I have a search showing the memory usage of an specific process of the server. Why am I doing this with Splunk and not with Nagios? Because in the JSON log we have detailed data of the tool running in the server which allows us to identify the UCase which triggered the event.

I would like to detect a 50% INCREASE of the memory usage in a period of 1 hour, and configure an ALERT to send an email to the admins.

Base search:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | timechart span=1m avg(extra.memory) AS Memory

I tried with diff, but I can't specify the 1h span, just saying the position:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | rename extra.memory as Memory|  diff pos1=1 pos2=500 attribute=Memory

JSON EXAMPLE - BEFORE

{ [-] 
    extra: { [-] 
      build_number:  3082 
      builder_name:  master-preintegration 
      duration:  2996 
      memory:  10876836
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  tlsisbld108lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-preintegration/builds/3082 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  54608 
    tags: [ 
   ] 
    timestamp:  2015-10-28 10:59:50,210 
    type:  Splunk 
}

JSON EXAMPLE - AFTER

{ [-] 
    extra: { [-] 
      build_number:  6670 
      builder_name:  master-engineering 
      duration:  2185 
      memory:  78950620 
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  shsisbld009lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-engineering/builds/6670 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  24041 
    tags: [ 
   ] 
    timestamp:  2015-10-28 11:17:11,264 
    type:  Splunk 
}
0 Karma
1 Solution

somesoni2
Revered Legend

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5

View solution in original post

0 Karma

somesoni2
Revered Legend

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5
0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...