Splunk Search

How to get a Splunk Alert if Value exceeds 90?

praneeth_lv
Observer

Hi We have a performance log onboarded and there is a value in that we would like to monitor:

The logs contain the following : 

{"name":"dbcp.numActive","value":"0"},

 

I would like to get an alert if the value is greater than 90 , how to i compile a query for this?

Labels (3)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @praneeth_lv,

I suppose that you already extracted firld from your data flow, so you could try to run something like this:

index=your_index name="dbcp.numActive"
| stats count
| where count>90

if you didn't extracted the field, you have to extract it:

index=your_index 
| rex "\{\"name\":\"(?<name>[^\"]+)"
| search name="dbcp.numActive"
| stats count
| where count>90

ciao.

Giuseppe

0 Karma

praneeth_lv
Observer

@gcusello @yuanliu 
Thanks for your inputs it didn't work.. 

The query i use is "sourcetype=log4j host="hostname*" source="/apps/application/data/log/app-app-perf.log" "dbcp.numActive"

I get the following result: We want only: {"name":"dbcp.numActive","value":"1"} from this output and alert when value is above 90 , 
2023-06-22 05:33:54,529 PLATFORMINSTRUMENTS {"timestamp":"1687426434","instrumentList":[{"name":"sr.jql-functions.linkedIssuesOf","value":"1077256"},{"name":"writer.lucene.commit","value":"42269"},{"name":"quicksearch.concurrent.search","value":"0"},{"name":"cache.i18n.CachingI18nFactory.size","value":"27"},{"name":"jmx.thread.cpu.time","value":"565716766328348"},{"name":"sr.jql-functions.commented","value":"1120"},{"name":"entity.users.total","value":"102471"},{"name":"issue.link.count","value":"5243"},{"name":"jmx.thread.cpu.wait.time","value":"0"},{"name":"sr.jql-functions.parentsOf","value":"4754"},{"name":"cache.i18n.CachingI18nFactory.loadSuccessCount","value":"0"},{"name":"entity.groups.total","value":"1135"},{"name":"db.reads","value":"330324205"},{"name":"five.hundreds","value":"1271"},{"name":"issue.search.count","value":"0"},{"name":"db.conns.borrowed","value":"2"},{"name":"cache.JiraOsgiContainerManager.loadSuccessCount","value":"0"},{"name":"jmx.thread.total.count","value":"991"},{"name":"db.writes","value":"4297749"},{"name":"cache.JiraOsgiContainerManager.missCount","value":"0"},{"name":"jmx.thread.peak.count","value":"1107"},{"name":"jmx.class.loaded.current","value":"183387"},{"name":"dashboard.view.count","value":"10561"},{"name":"cache.i18n.CachingI18nFactory.hitCount","value":"0"},{"name":"cache.i18n.CachingI18nFactory.totalLoadTime","value":"0"},{"name":"entity.workflows.total","value":"99"},{"name":"jmx.class.loaded.total","value":"204005"},{"name":"db.conns.time.to.borrow","value":"0"},{"name":"entity.attachments.total","value":"6389620"},{"name":"jmx.thread.cpu.wait.count","value":"0"},{"name":"issue.index.reads","value":"65206449"},{"name":"entity.projects.total","value":"2112"},{"name":"issue.worklogged.count","value":"2082"},{"name":"sr.jql-functions.addedAfterSprintStart","value":"87553"},{"name":"jira.license","value":"0"},{"name":"jmx.thread.ever.count","value":"222866"},{"name":"db.conns","value":"544273077"},{"name":"cache.i18n.CachingI18nFactory.missCount","value":"0"},{"name":"dbcp.maxActive","value":"-1"},{"name":"concurrent.requests","value":"1"},{"name":"jmx.memory.nonheap.committed","value":"2052964352"},{"name":"replicated.index.operations.total","value":"846969"},{"name":"sr.jql-functions.removedAfterSprintStart","value":"71708"},{"name":"dbcp.numIdle","value":"31"},{"name":"sr.jql-functions.releaseDate","value":"30233"},{"name":"sr.jql-functions.linkedIssuesOfAllRecursive","value":"1107"},{"name":"entity.versions.total","value":"77065"},{"name":"jmx.memory.nonheap.used","value":"1675480248"},{"name":"cache.VelocityTemplateCache.missCount","value":"0"},{"name":"cache.VelocityTemplateCache.directives.loadSuccessCount","value":"0"},{"name":"cache.JiraOsgiContainerManager.size","value":"24"},{"name":"entity.issues.total","value":"10993215"},{"name":"jmx.memory.heap.used","value":"19705760440"},{"name":"sr.jql-functions.epicsOf","value":"433667"},{"name":"sr.jql-functions.aggregateExpression","value":"7"},{"name":"cache.VelocityTemplateCache.loadSuccessCount","value":"0"},{"name":"sr.jql-functions.earliestUnreleasedVersionByReleaseDate","value":"96"},{"name":"sr.jql-functions.hasLinkType","value":"20"},{"name":"cache.VelocityTemplateCache.size","value":"324"},{"name":"issue.created.count","value":"4306"},{"name":"jmx.thread.nondaemon.count","value":"252"},{"name":"jmx.thread.daemon.count","value":"739"},{"name":"sr.jql-functions.overdue","value":"11332"},{"name":"http.session.objects","value":"4359"},{"name":"sr.jql-functions.hasLinks","value":"20093"},{"name":"cache.VelocityTemplateCache.directives.hitCount","value":"0"},{"name":"cache.i18n.CachingI18nFactory.loadExceptionCount","value":"0"},{"name":"dbcp.numActive","value":"1"},{"name":"http.sessions","value":"664"},{"name":"sr.jql-functions.issuesInEpics","value":"214293"},

0 Karma

yuanliu
SplunkTrust
SplunkTrust

Thank you for share the raw form of event.  In this case, you probably do not have the fields ready for use.  But extracting them is fairly easy with spath. and mvexpand once you cut out the conformant JSON for processing.

 

| eval json = replace(_raw, "^[\d:, -]+ \w+ {", "{")
| spath input=json path=instrumentList{}
| mvexpand instrumentList{}
| spath input=instrumentList{} ``` after this, you get a series of events with name and value as field names ```
| where name="dbcp.numActive" AND value > 90

 

Here is data emulation that you can play with and compare with real data. (I suppose the raw data is conformant and you did not list to the end of event.  So, I added a closing square bracket and a curly bracket.)

 

| makeresults
| eval _raw = "2023-06-22 05:33:54,529 PLATFORMINSTRUMENTS {\"timestamp\":\"1687426434\",\"instrumentList\":[{\"name\":\"sr.jql-functions.linkedIssuesOf\",\"value\":\"1077256\"},{\"name\":\"writer.lucene.commit\",\"value\":\"42269\"},{\"name\":\"quicksearch.concurrent.search\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.size\",\"value\":\"27\"},{\"name\":\"jmx.thread.cpu.time\",\"value\":\"565716766328348\"},{\"name\":\"sr.jql-functions.commented\",\"value\":\"1120\"},{\"name\":\"entity.users.total\",\"value\":\"102471\"},{\"name\":\"issue.link.count\",\"value\":\"5243\"},{\"name\":\"jmx.thread.cpu.wait.time\",\"value\":\"0\"},{\"name\":\"sr.jql-functions.parentsOf\",\"value\":\"4754\"},{\"name\":\"cache.i18n.CachingI18nFactory.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"entity.groups.total\",\"value\":\"1135\"},{\"name\":\"db.reads\",\"value\":\"330324205\"},{\"name\":\"five.hundreds\",\"value\":\"1271\"},{\"name\":\"issue.search.count\",\"value\":\"0\"},{\"name\":\"db.conns.borrowed\",\"value\":\"2\"},{\"name\":\"cache.JiraOsgiContainerManager.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"jmx.thread.total.count\",\"value\":\"991\"},{\"name\":\"db.writes\",\"value\":\"4297749\"},{\"name\":\"cache.JiraOsgiContainerManager.missCount\",\"value\":\"0\"},{\"name\":\"jmx.thread.peak.count\",\"value\":\"1107\"},{\"name\":\"jmx.class.loaded.current\",\"value\":\"183387\"},{\"name\":\"dashboard.view.count\",\"value\":\"10561\"},{\"name\":\"cache.i18n.CachingI18nFactory.hitCount\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.totalLoadTime\",\"value\":\"0\"},{\"name\":\"entity.workflows.total\",\"value\":\"99\"},{\"name\":\"jmx.class.loaded.total\",\"value\":\"204005\"},{\"name\":\"db.conns.time.to.borrow\",\"value\":\"0\"},{\"name\":\"entity.attachments.total\",\"value\":\"6389620\"},{\"name\":\"jmx.thread.cpu.wait.count\",\"value\":\"0\"},{\"name\":\"issue.index.reads\",\"value\":\"65206449\"},{\"name\":\"entity.projects.total\",\"value\":\"2112\"},{\"name\":\"issue.worklogged.count\",\"value\":\"2082\"},{\"name\":\"sr.jql-functions.addedAfterSprintStart\",\"value\":\"87553\"},{\"name\":\"jira.license\",\"value\":\"0\"},{\"name\":\"jmx.thread.ever.count\",\"value\":\"222866\"},{\"name\":\"db.conns\",\"value\":\"544273077\"},{\"name\":\"cache.i18n.CachingI18nFactory.missCount\",\"value\":\"0\"},{\"name\":\"dbcp.maxActive\",\"value\":\"-1\"},{\"name\":\"concurrent.requests\",\"value\":\"1\"},{\"name\":\"jmx.memory.nonheap.committed\",\"value\":\"2052964352\"},{\"name\":\"replicated.index.operations.total\",\"value\":\"846969\"},{\"name\":\"sr.jql-functions.removedAfterSprintStart\",\"value\":\"71708\"},{\"name\":\"dbcp.numIdle\",\"value\":\"31\"},{\"name\":\"sr.jql-functions.releaseDate\",\"value\":\"30233\"},{\"name\":\"sr.jql-functions.linkedIssuesOfAllRecursive\",\"value\":\"1107\"},{\"name\":\"entity.versions.total\",\"value\":\"77065\"},{\"name\":\"jmx.memory.nonheap.used\",\"value\":\"1675480248\"},{\"name\":\"cache.VelocityTemplateCache.missCount\",\"value\":\"0\"},{\"name\":\"cache.VelocityTemplateCache.directives.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"cache.JiraOsgiContainerManager.size\",\"value\":\"24\"},{\"name\":\"entity.issues.total\",\"value\":\"10993215\"},{\"name\":\"jmx.memory.heap.used\",\"value\":\"19705760440\"},{\"name\":\"sr.jql-functions.epicsOf\",\"value\":\"433667\"},{\"name\":\"sr.jql-functions.aggregateExpression\",\"value\":\"7\"},{\"name\":\"cache.VelocityTemplateCache.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"sr.jql-functions.earliestUnreleasedVersionByReleaseDate\",\"value\":\"96\"},{\"name\":\"sr.jql-functions.hasLinkType\",\"value\":\"20\"},{\"name\":\"cache.VelocityTemplateCache.size\",\"value\":\"324\"},{\"name\":\"issue.created.count\",\"value\":\"4306\"},{\"name\":\"jmx.thread.nondaemon.count\",\"value\":\"252\"},{\"name\":\"jmx.thread.daemon.count\",\"value\":\"739\"},{\"name\":\"sr.jql-functions.overdue\",\"value\":\"11332\"},{\"name\":\"http.session.objects\",\"value\":\"4359\"},{\"name\":\"sr.jql-functions.hasLinks\",\"value\":\"20093\"},{\"name\":\"cache.VelocityTemplateCache.directives.hitCount\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.loadExceptionCount\",\"value\":\"0\"},{\"name\":\"dbcp.numActive\",\"value\":\"1\"},{\"name\":\"http.sessions\",\"value\":\"664\"},{\"name\":\"sr.jql-functions.issuesInEpics\",\"value\":\"214293\"}]}"
``` data emulation above ```

 

 So, after the last spath, it gives me something like

namevalue
sr.jql-functions.linkedIssuesOf1077256
writer.lucene.commit42269
quicksearch.concurrent.search0
cache.i18n.CachingI18nFactory.size27
jmx.thread.cpu.time565716766328348
sr.jql-functions.commented1120
entity.users.total102471
issue.link.count5243
jmx.thread.cpu.wait.time0
sr.jql-functions.parentsOf4754
cache.i18n.CachingI18nFactory.loadSuccessCount0
entity.groups.total1135
db.reads330324205
five.hundreds1271
issue.search.count0
... 
Tags (2)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

As a rule, it is always helpful to illustrate complete raw events (in text).  In your illustration, is the JSON the complete log or one node of a larger JSON?  If it is the complete JSON, Splunk would have given you two fields, "name" and "value".  I assume that you want to alert when name has the value "dbcp.numActive", not just any value.  So, this should suffice

<any other criteria> name="dbcp.numActive" value > 90

Does this help?

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...