How to compare a json value field in a 1h window, and detect a 50% increase in the value

[matiasburni]

Hi all,

I have a search showing the memory usage of an specific process of the server. Why am I doing this with Splunk and not with Nagios? Because in the JSON log we have detailed data of the tool running in the server which allows us to identify the UCase which triggered the event.

I would like to detect a 50% INCREASE of the memory usage in a period of 1 hour, and configure an ALERT to send an email to the admins.

Base search:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | timechart span=1m avg(extra.memory) AS Memory

I tried with diff, but I can't specify the 1h span, just saying the position:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | rename extra.memory as Memory|  diff pos1=1 pos2=500 attribute=Memory

JSON EXAMPLE - BEFORE

{ [-] 
    extra: { [-] 
      build_number:  3082 
      builder_name:  master-preintegration 
      duration:  2996 
      memory:  10876836
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  tlsisbld108lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-preintegration/builds/3082 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  54608 
    tags: [ 
   ] 
    timestamp:  2015-10-28 10:59:50,210 
    type:  Splunk 
}

JSON EXAMPLE - AFTER

{ [-] 
    extra: { [-] 
      build_number:  6670 
      builder_name:  master-engineering 
      duration:  2185 
      memory:  78950620 
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  shsisbld009lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-engineering/builds/6670 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  24041 
    tags: [ 
   ] 
    timestamp:  2015-10-28 11:17:11,264 
    type:  Splunk 
}

[somesoni2 · Accepted solution]

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5