Alerting

How to compare a json value field in a 1h window, and detect a 50% increase in the value

matiasburni
Engager

Hi all,

I have a search showing the memory usage of an specific process of the server. Why am I doing this with Splunk and not with Nagios? Because in the JSON log we have detailed data of the tool running in the server which allows us to identify the UCase which triggered the event.

I would like to detect a 50% INCREASE of the memory usage in a period of 1 hour, and configure an ALERT to send an email to the admins.

Base search:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | timechart span=1m avg(extra.memory) AS Memory

I tried with diff, but I can't specify the 1h span, just saying the position:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | rename extra.memory as Memory|  diff pos1=1 pos2=500 attribute=Memory

JSON EXAMPLE - BEFORE

{ [-] 
    extra: { [-] 
      build_number:  3082 
      builder_name:  master-preintegration 
      duration:  2996 
      memory:  10876836
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  tlsisbld108lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-preintegration/builds/3082 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  54608 
    tags: [ 
   ] 
    timestamp:  2015-10-28 10:59:50,210 
    type:  Splunk 
}

JSON EXAMPLE - AFTER

{ [-] 
    extra: { [-] 
      build_number:  6670 
      builder_name:  master-engineering 
      duration:  2185 
      memory:  78950620 
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  shsisbld009lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-engineering/builds/6670 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  24041 
    tags: [ 
   ] 
    timestamp:  2015-10-28 11:17:11,264 
    type:  Splunk 
}
0 Karma
1 Solution

somesoni2
Revered Legend

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5

View solution in original post

0 Karma

somesoni2
Revered Legend

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5
0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...