All Apps and Add-ons

REST API JSON Parsing & Event breaking issue

ashish9433
Communicator

Hi,

I have tried Splunk Add-on builder as well as REST API app of App store to get data via REST API from Mongo DB OPS Manager, but the results events are not getting broken properly

I tried fiddling dump, dumps, load & loads function in python as well as whatever i could think of in props.conf but no positive results.

Any inputs on how do i fix it?

Below is how i see the data in splunk irrespective i use Splunk Add-on builder or REST API App

{"links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts?status=OPEN&pageNum=1&itemsPerPage=100","rel":"self"}],"results":[{"acknowledgedUntil":"2119-03-01T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bcc00e545f881d3659a1","created":"2018-08-21T14:49:27Z","currentValue":{"number":2.3656959E7,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5b7c26771f98cf1493f1577b","lastNotified":"2019-03-25T01:24:51Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5b7c26771f98cf1493f1577b","rel":"self"}],"metricName":"OPLOG_SLAVE_LAG_MASTER_TIME","replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-05-22T09:52:52Z"},{"acknowledgedUntil":"2119-06-22T14:08:19Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf8a0e545f881d3663fd","created":"2019-03-09T23:20:40Z","eventTypeName":"HOST_RECOVERING","groupId":"5a0dbc440e545f72701d8ca6","hostId":"8d9e4f32e307e1b4a935d0f3e0055940","hostnameAndPort":"denata3utmdb01.abc.xyz.org:27045","id":"5c844a480e545f1d153be335","lastNotified":"2019-05-21T21:24:59Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c844a480e545f1d153be335","rel":"self"}],"replicaSetName":"ATBDAP03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:19Z"},{"acknowledgedUntil":"2119-06-22T14:08:09Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980c0390e545f881d366688","created":"2019-03-29T01:49:16Z","eventTypeName":"HOST_DOWN","groupId":"5994baea0e545f0fb11a7bf8","hostId":"61a2829f7bbb5f89f3406642206a36fa","hostnameAndPort":"denatb3mdips01.abc.xyz.org:27045","id":"5c9d799cf76e9d3f9d9934f1","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5c9d799cf76e9d3f9d9934f1","rel":"self"}],"replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST","updated":"2019-07-16T14:08:09Z"},{"acknowledgedUntil":"2119-06-22T14:08:03Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-04-26T01:48:10Z","currentValue":{"number":92.42220000199045,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4711110d008f035c85bc9edd549bb46","hostnameAndPort":"denatb3mdips02.abc.xyz.org:27045","id":"5cc2635a0e545f54c4245073","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cc2635a0e545f54c4245073","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:08:03Z"},{"acknowledgedUntil":"2119-06-22T14:07:58Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-03T17:17:46Z","currentValue":{"number":93.33982523912108,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"b4270e868d403038d18dad42693416f5","hostnameAndPort":"denatb3mdips03.abc.xyz.org:27045","id":"5ccc77ba0e545f54c4d931d4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ccc77ba0e545f54c4d931d4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ATIPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:58Z"},{"acknowledgedUntil":"2119-06-22T14:07:52Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bf280e545f881d36629d","created":"2019-05-08T01:46:28Z","currentValue":{"number":91.2596940644846,"units":"RAW"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"5994baea0e545f0fb11a7bf8","hostId":"697b2c7697e9fb5f466a6f6f65d1f914","hostnameAndPort":"dens2b3mdips02.abc.xyz.org:27045","id":"5cd234f4f76e9d580f1c16b4","lastNotified":"2019-05-21T21:24:53Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5cd234f4f76e9d580f1c16b4","rel":"self"}],"metricName":"DISK_PARTITION_SPACE_USED_DATA","replicaSetName":"ST2IPS03","status":"OPEN","tags":["TAG_SDLC"],"typeName":"HOST_METRIC","updated":"2019-07-16T14:07:52Z"},{"acknowledgedUntil":"2119-06-22T14:07:43Z","acknowledgingUsername":"abc@zyz.com","alertConfigId":"5980bd950e545f881d365cb2","created":"2019-05-20T14:35:51Z","currentValue":{"number":155845.0,"units":"SECONDS"},"eventTypeName":"OUTSIDE_METRIC_THRESHOLD","groupId":"597f93d70e545f881d331591","hostId":"42d7c182422a623b875df934da1cc3e9","hostnameAndPort":"deni1b3mdedi01.abc.xyz.org:27130","id":"5ce2bb470e545f55d16ed5b4","lastNotified":"2019-05-22T02:39:58Z","links":[{"href":"http://opsmanager.abc.xyz.org:80/api/public/v1.0/globalAlerts/5ce2bb470e545f55d16ed5b4","rel":"self"}]
0 Karma
1 Solution

woodcock
Esteemed Legend

You are overkilling this effort. Just do this:

[<your sourcetype here>]
LINK_BREAKER = <Your RegEx here>
SHOULD_LINEMERGE = false
KV_MODE = json

Don't bother with the tools; just get the LINE_BREAKER RegEx right and you are good-to-go.

View solution in original post

0 Karma

woodcock
Esteemed Legend

You are overkilling this effort. Just do this:

[<your sourcetype here>]
LINK_BREAKER = <Your RegEx here>
SHOULD_LINEMERGE = false
KV_MODE = json

Don't bother with the tools; just get the LINE_BREAKER RegEx right and you are good-to-go.

0 Karma

ashish9433
Communicator

Yup baiscally the rest API was sending out the data is random order everytime it polled and the regex was written without that things to be taken into consideration. Later on it was realized that the JSON response is completely jumbled up on every hit and fixing the Regex solved the prblm.

Thanks for your help!

0 Karma

chli_splunk
Splunk Employee
Splunk Employee

In Add-on Builder, you can input a jsonpath to break a list object into events. Based on your example, you can simply input "$results" in Add-on Builder -> your REST input -> Event extraction settings -> JSON path. And then we can preview the events after clicking Test button.

0 Karma

ashish9433
Communicator

This doesnt work and in preview even with/without the events shows properly formatted and parsed, but i don't know why when it is getting indexed it is messing up

0 Karma

chli_splunk
Splunk Employee
Splunk Employee

I'm surprised it's not working. Maybe other configurations may affect the line breaker but I cannot tell without details.
Please make sure the response is valid JSON. What you posted here should end with "}"?

0 Karma

ashish9433
Communicator

yeah even i am banging my head that what is that i am missing. I have posted only half of the output. The actual output is a valid json.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...