topic Re: How to parse a JSON from REST API? in Getting Data In

How to parse a JSON from REST API?

claudio_palmeri — Fri, 06 Jul 2018 12:48:47 GMT

Hi all,
I need some help parsing a JSON containing none/one/multiple nested messages that I have imported via REST API (poll). I am saying one or multiple or none as it depends on what the poll is retrieving from the REST API.

In the event that the poll is retrieving no new events, I would like Splunk not to show an empty entry (square brackets only) in the log. In the event of a single message, I'd like to remove the header/footer (square brackets) and display the event. In the event of multiple messages, I would like Splunk to remove header/footer as well as split the message in individual events, with associated timestamp.

I have been doing a lot of research, however I am struggling with the various attributes in $SPLUNK_HOME/etc/system/local/props.conf and with the related regular expressions to make the whole thing work.

A sample JSON with three messages is below.

[
    {
        "source": "APPNAME",
        "id": "1234567890",
        "recorded": "2018-06-07T00:44:22.584Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email1@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message1"
        }
    },
    {
        "source": "APPNAME",
        "id": "2345678901",
        "recorded": "2018-05-07T05:12:47.409Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email2@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "POLICY",
            "message": "Details:\nIP Address: 222.222.222.222\nCountry: AU\nNew Device: true \"Default Action\"\n"
        }
    },
    {
        "source": "APPNAME",
        "id": "3456789012",
        "recorded": "2018-05-07T05:12:58.137Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email3@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message3"
        }
    }   
]

I am not even going to bother showing what I have put in the props.conf file as it might be misleading.

I am facing this problem only when I am polling the JSON via REST API. In comparison, if I get the same JSON (flat file) and manually import it, Splunk is smart enough to break it down cleanly in individual events.

Any assistance would be highly appreciated. Thank you.

Re: How to parse a JSON from REST API?

pruthvikrishnap — Fri, 06 Jul 2018 19:09:00 GMT

Hi,

This can be handled by creating a custome responsehandler i.e by editing responsehandler.py (/rest/bin/responsehandlers.py)

or you can exclude null values by adding below to your search.

search | where isnull(field name)

Let me know if this helps.

Re: How to parse a JSON from REST API?

claudio_palmeri — Tue, 29 Sep 2020 20:21:08 GMT

Hi pruthvikrishnapolavarapu,
It does help, but I wouldn't know where to start to be honest. Is this something that you can guide me through or perhaps can you recommend a Knowledge Article to follow?

Should I use the responsehandler.py would I also need to modify props.conf at all? At the moment I have a combination of SEDCMD-remove_header, SEDCMD-remove_footer, LINE_BREAKER and TIME_PREFIX that are partially doing the job, but the REGEX need some fixing...

Re: How to parse a JSON from REST API?

claudio_palmeri — Fri, 06 Jul 2018 22:22:30 GMT

I've seen this:

https://github.com/damiendallimore/SplunkModularInputsPythonFramework/blob/master/implementations/rest/bin/responsehandlers.py

and this

https://answers.splunk.com/answers/233620/how-to-use-custom-response-handlers-for-monitoring-1.html

but again, I wouldn't know where to start to configure them for my exact scenario...

Re: How to parse a JSON from REST API?

claudio_palmeri — Sat, 07 Jul 2018 12:02:55 GMT

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...