REST API JSON Parsing & Event breaking issue

[ashish9433]

Hi,

I have tried Splunk Add-on builder as well as REST API app of App store to get data via REST API from Mongo DB OPS Manager, but the results events are not getting broken properly

I tried fiddling dump, dumps, load & loads function in python as well as whatever i could think of in props.conf but no positive results.

Any inputs on how do i fix it?

Below is how i see the data in splunk irrespective i use Splunk Add-on builder or REST API App

{"links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts?status=OPEN&pageNum=1&itemsPerPage=100","rel":"self"}],"results":[{"acknowledgedUntil":"2119-03-01T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bcc00e545f881d3659a1","created":"2018-08-21T14:49:27Z","currentValue":{"number":2.3656959E7,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5b7c26771f98cf1493f1577b","lastNotified":"2019-03-25T01:24:51Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5b7c26771f98cf1493f1577b","rel":"self"}],"metricName":"OPLOG_SLAVE_LAG_MASTER_TIME","replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-05-22T09:52:52Z"},{"acknowledgedUntil":"2119-06-22T14:08:19Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf8a0e545f881d3663fd","created":"2019-03-09T23:20:40Z","eventTypeName":"HOST_RECOVERING","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5c844a480e545f1d153be335","lastNotified":"2019-05-21T21:24:59Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c844a480e545f1d153be335","rel":"self"}],"replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:19Z"},{"acknowledgedUntil":"2119-06-22T14:08:09Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980c0390e545f881d366688","created":"2019-03-29T01:49:16Z","eventTypeName":"HOST_DOWN","groupId":"5994baea0e545f0fb11a7bf8","hostId":"61a2829f7bbb5f89f3406642206a36fa","hostnameAndPort":"denatb3mdips01.abc.xyz.org:27045","id":"5c9d799cf76e9d3f9d9934f1","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c9d799cf76e9d3f9d9934f1","rel":"self"}],"replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:09Z"},{"acknowledgedUntil":"2119-06-22T14:08:03Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-04-26T01:48:10Z","currentValue":{"number":92.42220000199045,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4711110d008f035c85bc9edd549bb46","hostnameAndPort":"denatb3mdips02.abc.xyz.org:27045","id":"5cc2635a0e545f54c4245073","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cc2635a0e545f54c4245073","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:08:03Z"},{"acknowledgedUntil":"2119-06-22T14:07:58Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-03T17:17:46Z","currentValue":{"number":93.33982523912108,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4270e868d403038d18dad42693416f5","hostnameAndPort":"denatb3mdips03.abc.xyz.org:27045","id":"5ccc77ba0e545f54c4d931d4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ccc77ba0e545f54c4d931d4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:58Z"},{"acknowledgedUntil":"2119-06-22T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-08T01:46:28Z","currentValue":{"number":91.2596940644846,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"697b2c7697e9fb5f466a6f6f65d1f914","hostnameAndPort":"dens2b3mdips02.abc.xyz.org:27045","id":"5cd234f4f76e9d580f1c16b4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cd234f4f76e9d580f1c16b4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ST2IPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:52Z"},{"acknowledgedUntil":"2119-06-22T14:07:43Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bd950e545f881d365cb2","created":"2019-05-20T14:35:51Z","currentValue":{"number":155845.0,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"597f93d70e545f881d331591","hostId":"42d7c182422a623b875df934da1cc3e9","hostnameAndPort":"deni1b3mdedi01.abc.xyz.org:27130","id":"5ce2bb470e545f55d16ed5b4","lastNotified":"2019-05-22T02:39:58Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ce2bb470e545f55d16ed5b4","rel":"self"}]

[woodcock · Accepted solution]

You are overkilling this effort. Just do this:

[<your sourcetype here>]
LINK_BREAKER = <Your RegEx here>
SHOULD_LINEMERGE = false
KV_MODE = json

Don't bother with the tools; just get the LINE_BREAKER RegEx right and you are good-to-go.

[ashish9433]

Yup baiscally the rest API was sending out the data is random order everytime it polled and the regex was written without that things to be taken into consideration. Later on it was realized that the JSON response is completely jumbled up on every hit and fixing the Regex solved the prblm.

Thanks for your help!

[chli_splunk]

In Add-on Builder, you can input a jsonpath to break a list object into events. Based on your example, you can simply input "$results" in Add-on Builder -> your REST input -> Event extraction settings -> JSON path. And then we can preview the events after clicking Test button.

[ashish9433]

This doesnt work and in preview even with/without the events shows properly formatted and parsed, but i don't know why when it is getting indexed it is messing up

[chli_splunk]

I'm surprised it's not working. Maybe other configurations may affect the line breaker but I cannot tell without details.
Please make sure the response is valid JSON. What you posted here should end with "}"?

[ashish9433]

yeah even i am banging my head that what is that i am missing. I have posted only half of the output. The actual output is a valid json.