All Apps and Add-ons

REST API JSON Parsing & Event breaking issue

ashish9433
Communicator

Hi,

I have tried Splunk Add-on builder as well as REST API app of App store to get data via REST API from Mongo DB OPS Manager, but the results events are not getting broken properly

I tried fiddling dump, dumps, load & loads function in python as well as whatever i could think of in props.conf but no positive results.

Any inputs on how do i fix it?

Below is how i see the data in splunk irrespective i use Splunk Add-on builder or REST API App

{"links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts?status=OPEN&pageNum=1&itemsPerPage=100","rel":"self"}],"results":[{"acknowledgedUntil":"2119-03-01T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bcc00e545f881d3659a1","created":"2018-08-21T14:49:27Z","currentValue":{"number":2.3656959E7,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5b7c26771f98cf1493f1577b","lastNotified":"2019-03-25T01:24:51Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5b7c26771f98cf1493f1577b","rel":"self"}],"metricName":"OPLOG_SLAVE_LAG_MASTER_TIME","replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-05-22T09:52:52Z"},{"acknowledgedUntil":"2119-06-22T14:08:19Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf8a0e545f881d3663fd","created":"2019-03-09T23:20:40Z","eventTypeName":"HOST_RECOVERING","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5c844a480e545f1d153be335","lastNotified":"2019-05-21T21:24:59Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c844a480e545f1d153be335","rel":"self"}],"replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:19Z"},{"acknowledgedUntil":"2119-06-22T14:08:09Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980c0390e545f881d366688","created":"2019-03-29T01:49:16Z","eventTypeName":"HOST_DOWN","groupId":"5994baea0e545f0fb11a7bf8","hostId":"61a2829f7bbb5f89f3406642206a36fa","hostnameAndPort":"denatb3mdips01.abc.xyz.org:27045","id":"5c9d799cf76e9d3f9d9934f1","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c9d799cf76e9d3f9d9934f1","rel":"self"}],"replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:09Z"},{"acknowledgedUntil":"2119-06-22T14:08:03Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-04-26T01:48:10Z","currentValue":{"number":92.42220000199045,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4711110d008f035c85bc9edd549bb46","hostnameAndPort":"denatb3mdips02.abc.xyz.org:27045","id":"5cc2635a0e545f54c4245073","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cc2635a0e545f54c4245073","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:08:03Z"},{"acknowledgedUntil":"2119-06-22T14:07:58Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-03T17:17:46Z","currentValue":{"number":93.33982523912108,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4270e868d403038d18dad42693416f5","hostnameAndPort":"denatb3mdips03.abc.xyz.org:27045","id":"5ccc77ba0e545f54c4d931d4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ccc77ba0e545f54c4d931d4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:58Z"},{"acknowledgedUntil":"2119-06-22T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-08T01:46:28Z","currentValue":{"number":91.2596940644846,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"697b2c7697e9fb5f466a6f6f65d1f914","hostnameAndPort":"dens2b3mdips02.abc.xyz.org:27045","id":"5cd234f4f76e9d580f1c16b4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cd234f4f76e9d580f1c16b4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ST2IPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:52Z"},{"acknowledgedUntil":"2119-06-22T14:07:43Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bd950e545f881d365cb2","created":"2019-05-20T14:35:51Z","currentValue":{"number":155845.0,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"597f93d70e545f881d331591","hostId":"42d7c182422a623b875df934da1cc3e9","hostnameAndPort":"deni1b3mdedi01.abc.xyz.org:27130","id":"5ce2bb470e545f55d16ed5b4","lastNotified":"2019-05-22T02:39:58Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ce2bb470e545f55d16ed5b4","rel":"self"}]
0 Karma
1 Solution

woodcock
Esteemed Legend

You are overkilling this effort. Just do this:

[<your sourcetype here>]
LINK_BREAKER = <Your RegEx here>
SHOULD_LINEMERGE = false
KV_MODE = json

Don't bother with the tools; just get the LINE_BREAKER RegEx right and you are good-to-go.

View solution in original post

0 Karma

woodcock
Esteemed Legend

You are overkilling this effort. Just do this:

[<your sourcetype here>]
LINK_BREAKER = <Your RegEx here>
SHOULD_LINEMERGE = false
KV_MODE = json

Don't bother with the tools; just get the LINE_BREAKER RegEx right and you are good-to-go.

0 Karma

ashish9433
Communicator

Yup baiscally the rest API was sending out the data is random order everytime it polled and the regex was written without that things to be taken into consideration. Later on it was realized that the JSON response is completely jumbled up on every hit and fixing the Regex solved the prblm.

Thanks for your help!

0 Karma

chli_splunk
Splunk Employee
Splunk Employee

In Add-on Builder, you can input a jsonpath to break a list object into events. Based on your example, you can simply input "$results" in Add-on Builder -> your REST input -> Event extraction settings -> JSON path. And then we can preview the events after clicking Test button.

0 Karma

ashish9433
Communicator

This doesnt work and in preview even with/without the events shows properly formatted and parsed, but i don't know why when it is getting indexed it is messing up

0 Karma

chli_splunk
Splunk Employee
Splunk Employee

I'm surprised it's not working. Maybe other configurations may affect the line breaker but I cannot tell without details.
Please make sure the response is valid JSON. What you posted here should end with "}"?

0 Karma

ashish9433
Communicator

yeah even i am banging my head that what is that i am missing. I have posted only half of the output. The actual output is a valid json.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

&#x1f342; Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...