How to get a Splunk Alert if Value exceeds 90?

[praneeth_lv]

Hi We have a performance log onboarded and there is a value in that we would like to monitor:

The logs contain the following : 

{"name":"dbcp.numActive","value":"0"},

I would like to get an alert if the value is greater than 90 , how to i compile a query for this?

[gcusello]

Hi @praneeth_lv,

I suppose that you already extracted firld from your data flow, so you could try to run something like this:

index=your_index name="dbcp.numActive"
| stats count
| where count>90

if you didn't extracted the field, you have to extract it:

index=your_index 
| rex "\{\"name\":\"(?<name>[^\"]+)"
| search name="dbcp.numActive"
| stats count
| where count>90

ciao.

Giuseppe

[praneeth_lv]

@gcusello @yuanliu 
Thanks for your inputs it didn't work.. 

The query i use is "sourcetype=log4j host="hostname*" source="/apps/application/data/log/app-app-perf.log" "dbcp.numActive"

I get the following result: We want only: {"name":"dbcp.numActive","value":"1"} from this output and alert when value is above 90 , 
2023-06-22 05:33:54,529 PLATFORMINSTRUMENTS {"timestamp":"1687426434","instrumentList":[{"name":"sr.jql-functions.linkedIssuesOf","value":"1077256"},{"name":"writer.lucene.commit","value":"42269"},{"name":"quicksearch.concurrent.search","value":"0"},{"name":"cache.i18n.CachingI18nFactory.size","value":"27"},{"name":"jmx.thread.cpu.time","value":"565716766328348"},{"name":"sr.jql-functions.commented","value":"1120"},{"name":"entity.users.total","value":"102471"},{"name":"issue.link.count","value":"5243"},{"name":"jmx.thread.cpu.wait.time","value":"0"},{"name":"sr.jql-functions.parentsOf","value":"4754"},{"name":"cache.i18n.CachingI18nFactory.loadSuccessCount","value":"0"},{"name":"entity.groups.total","value":"1135"},{"name":"db.reads","value":"330324205"},{"name":"five.hundreds","value":"1271"},{"name":"issue.search.count","value":"0"},{"name":"db.conns.borrowed","value":"2"},{"name":"cache.JiraOsgiContainerManager.loadSuccessCount","value":"0"},{"name":"jmx.thread.total.count","value":"991"},{"name":"db.writes","value":"4297749"},{"name":"cache.JiraOsgiContainerManager.missCount","value":"0"},{"name":"jmx.thread.peak.count","value":"1107"},{"name":"jmx.class.loaded.current","value":"183387"},{"name":"dashboard.view.count","value":"10561"},{"name":"cache.i18n.CachingI18nFactory.hitCount","value":"0"},{"name":"cache.i18n.CachingI18nFactory.totalLoadTime","value":"0"},{"name":"entity.workflows.total","value":"99"},{"name":"jmx.class.loaded.total","value":"204005"},{"name":"db.conns.time.to.borrow","value":"0"},{"name":"entity.attachments.total","value":"6389620"},{"name":"jmx.thread.cpu.wait.count","value":"0"},{"name":"issue.index.reads","value":"65206449"},{"name":"entity.projects.total","value":"2112"},{"name":"issue.worklogged.count","value":"2082"},{"name":"sr.jql-functions.addedAfterSprintStart","value":"87553"},{"name":"jira.license","value":"0"},{"name":"jmx.thread.ever.count","value":"222866"},{"name":"db.conns","value":"544273077"},{"name":"cache.i18n.CachingI18nFactory.missCount","value":"0"},{"name":"dbcp.maxActive","value":"-1"},{"name":"concurrent.requests","value":"1"},{"name":"jmx.memory.nonheap.committed","value":"2052964352"},{"name":"replicated.index.operations.total","value":"846969"},{"name":"sr.jql-functions.removedAfterSprintStart","value":"71708"},{"name":"dbcp.numIdle","value":"31"},{"name":"sr.jql-functions.releaseDate","value":"30233"},{"name":"sr.jql-functions.linkedIssuesOfAllRecursive","value":"1107"},{"name":"entity.versions.total","value":"77065"},{"name":"jmx.memory.nonheap.used","value":"1675480248"},{"name":"cache.VelocityTemplateCache.missCount","value":"0"},{"name":"cache.VelocityTemplateCache.directives.loadSuccessCount","value":"0"},{"name":"cache.JiraOsgiContainerManager.size","value":"24"},{"name":"entity.issues.total","value":"10993215"},{"name":"jmx.memory.heap.used","value":"19705760440"},{"name":"sr.jql-functions.epicsOf","value":"433667"},{"name":"sr.jql-functions.aggregateExpression","value":"7"},{"name":"cache.VelocityTemplateCache.loadSuccessCount","value":"0"},{"name":"sr.jql-functions.earliestUnreleasedVersionByReleaseDate","value":"96"},{"name":"sr.jql-functions.hasLinkType","value":"20"},{"name":"cache.VelocityTemplateCache.size","value":"324"},{"name":"issue.created.count","value":"4306"},{"name":"jmx.thread.nondaemon.count","value":"252"},{"name":"jmx.thread.daemon.count","value":"739"},{"name":"sr.jql-functions.overdue","value":"11332"},{"name":"http.session.objects","value":"4359"},{"name":"sr.jql-functions.hasLinks","value":"20093"},{"name":"cache.VelocityTemplateCache.directives.hitCount","value":"0"},{"name":"cache.i18n.CachingI18nFactory.loadExceptionCount","value":"0"},{"name":"dbcp.numActive","value":"1"},{"name":"http.sessions","value":"664"},{"name":"sr.jql-functions.issuesInEpics","value":"214293"},

[yuanliu]

Thank you for share the raw form of event.  In this case, you probably do not have the fields ready for use.  But extracting them is fairly easy with spath. and mvexpand once you cut out the conformant JSON for processing.

| eval json = replace(_raw, "^[\d:, -]+ \w+ {", "{")
| spath input=json path=instrumentList{}
| mvexpand instrumentList{}
| spath input=instrumentList{} ``` after this, you get a series of events with name and value as field names ```
| where name="dbcp.numActive" AND value > 90

Here is data emulation that you can play with and compare with real data. (I suppose the raw data is conformant and you did not list to the end of event.  So, I added a closing square bracket and a curly bracket.)

| makeresults
| eval _raw = "2023-06-22 05:33:54,529 PLATFORMINSTRUMENTS {\"timestamp\":\"1687426434\",\"instrumentList\":[{\"name\":\"sr.jql-functions.linkedIssuesOf\",\"value\":\"1077256\"},{\"name\":\"writer.lucene.commit\",\"value\":\"42269\"},{\"name\":\"quicksearch.concurrent.search\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.size\",\"value\":\"27\"},{\"name\":\"jmx.thread.cpu.time\",\"value\":\"565716766328348\"},{\"name\":\"sr.jql-functions.commented\",\"value\":\"1120\"},{\"name\":\"entity.users.total\",\"value\":\"102471\"},{\"name\":\"issue.link.count\",\"value\":\"5243\"},{\"name\":\"jmx.thread.cpu.wait.time\",\"value\":\"0\"},{\"name\":\"sr.jql-functions.parentsOf\",\"value\":\"4754\"},{\"name\":\"cache.i18n.CachingI18nFactory.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"entity.groups.total\",\"value\":\"1135\"},{\"name\":\"db.reads\",\"value\":\"330324205\"},{\"name\":\"five.hundreds\",\"value\":\"1271\"},{\"name\":\"issue.search.count\",\"value\":\"0\"},{\"name\":\"db.conns.borrowed\",\"value\":\"2\"},{\"name\":\"cache.JiraOsgiContainerManager.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"jmx.thread.total.count\",\"value\":\"991\"},{\"name\":\"db.writes\",\"value\":\"4297749\"},{\"name\":\"cache.JiraOsgiContainerManager.missCount\",\"value\":\"0\"},{\"name\":\"jmx.thread.peak.count\",\"value\":\"1107\"},{\"name\":\"jmx.class.loaded.current\",\"value\":\"183387\"},{\"name\":\"dashboard.view.count\",\"value\":\"10561\"},{\"name\":\"cache.i18n.CachingI18nFactory.hitCount\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.totalLoadTime\",\"value\":\"0\"},{\"name\":\"entity.workflows.total\",\"value\":\"99\"},{\"name\":\"jmx.class.loaded.total\",\"value\":\"204005\"},{\"name\":\"db.conns.time.to.borrow\",\"value\":\"0\"},{\"name\":\"entity.attachments.total\",\"value\":\"6389620\"},{\"name\":\"jmx.thread.cpu.wait.count\",\"value\":\"0\"},{\"name\":\"issue.index.reads\",\"value\":\"65206449\"},{\"name\":\"entity.projects.total\",\"value\":\"2112\"},{\"name\":\"issue.worklogged.count\",\"value\":\"2082\"},{\"name\":\"sr.jql-functions.addedAfterSprintStart\",\"value\":\"87553\"},{\"name\":\"jira.license\",\"value\":\"0\"},{\"name\":\"jmx.thread.ever.count\",\"value\":\"222866\"},{\"name\":\"db.conns\",\"value\":\"544273077\"},{\"name\":\"cache.i18n.CachingI18nFactory.missCount\",\"value\":\"0\"},{\"name\":\"dbcp.maxActive\",\"value\":\"-1\"},{\"name\":\"concurrent.requests\",\"value\":\"1\"},{\"name\":\"jmx.memory.nonheap.committed\",\"value\":\"2052964352\"},{\"name\":\"replicated.index.operations.total\",\"value\":\"846969\"},{\"name\":\"sr.jql-functions.removedAfterSprintStart\",\"value\":\"71708\"},{\"name\":\"dbcp.numIdle\",\"value\":\"31\"},{\"name\":\"sr.jql-functions.releaseDate\",\"value\":\"30233\"},{\"name\":\"sr.jql-functions.linkedIssuesOfAllRecursive\",\"value\":\"1107\"},{\"name\":\"entity.versions.total\",\"value\":\"77065\"},{\"name\":\"jmx.memory.nonheap.used\",\"value\":\"1675480248\"},{\"name\":\"cache.VelocityTemplateCache.missCount\",\"value\":\"0\"},{\"name\":\"cache.VelocityTemplateCache.directives.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"cache.JiraOsgiContainerManager.size\",\"value\":\"24\"},{\"name\":\"entity.issues.total\",\"value\":\"10993215\"},{\"name\":\"jmx.memory.heap.used\",\"value\":\"19705760440\"},{\"name\":\"sr.jql-functions.epicsOf\",\"value\":\"433667\"},{\"name\":\"sr.jql-functions.aggregateExpression\",\"value\":\"7\"},{\"name\":\"cache.VelocityTemplateCache.loadSuccessCount\",\"value\":\"0\"},{\"name\":\"sr.jql-functions.earliestUnreleasedVersionByReleaseDate\",\"value\":\"96\"},{\"name\":\"sr.jql-functions.hasLinkType\",\"value\":\"20\"},{\"name\":\"cache.VelocityTemplateCache.size\",\"value\":\"324\"},{\"name\":\"issue.created.count\",\"value\":\"4306\"},{\"name\":\"jmx.thread.nondaemon.count\",\"value\":\"252\"},{\"name\":\"jmx.thread.daemon.count\",\"value\":\"739\"},{\"name\":\"sr.jql-functions.overdue\",\"value\":\"11332\"},{\"name\":\"http.session.objects\",\"value\":\"4359\"},{\"name\":\"sr.jql-functions.hasLinks\",\"value\":\"20093\"},{\"name\":\"cache.VelocityTemplateCache.directives.hitCount\",\"value\":\"0\"},{\"name\":\"cache.i18n.CachingI18nFactory.loadExceptionCount\",\"value\":\"0\"},{\"name\":\"dbcp.numActive\",\"value\":\"1\"},{\"name\":\"http.sessions\",\"value\":\"664\"},{\"name\":\"sr.jql-functions.issuesInEpics\",\"value\":\"214293\"}]}"
``` data emulation above ```

 So, after the last spath, it gives me something like

name	value
sr.jql-functions.linkedIssuesOf	1077256
writer.lucene.commit	42269
quicksearch.concurrent.search	0
cache.i18n.CachingI18nFactory.size	27
jmx.thread.cpu.time	565716766328348
sr.jql-functions.commented	1120
entity.users.total	102471
issue.link.count	5243
jmx.thread.cpu.wait.time	0
sr.jql-functions.parentsOf	4754
cache.i18n.CachingI18nFactory.loadSuccessCount	0
entity.groups.total	1135
db.reads	330324205
five.hundreds	1271
issue.search.count	0
...

[yuanliu]

As a rule, it is always helpful to illustrate complete raw events (in text).  In your illustration, is the JSON the complete log or one node of a larger JSON?  If it is the complete JSON, Splunk would have given you two fields, "name" and "value".  I assume that you want to alert when name has the value "dbcp.numActive", not just any value.  So, this should suffice

<any other criteria> name="dbcp.numActive" value > 90

Does this help?