How to parse a JSON from REST API?

[claudio_palmeri]

Hi all,
I need some help parsing a JSON containing none/one/multiple nested messages that I have imported via REST API (poll). I am saying one or multiple or none as it depends on what the poll is retrieving from the REST API.

In the event that the poll is retrieving no new events, I would like Splunk not to show an empty entry (square brackets only) in the log. In the event of a single message, I'd like to remove the header/footer (square brackets) and display the event. In the event of multiple messages, I would like Splunk to remove header/footer as well as split the message in individual events, with associated timestamp.

I have been doing a lot of research, however I am struggling with the various attributes in $SPLUNK_HOME/etc/system/local/props.conf and with the related regular expressions to make the whole thing work.

A sample JSON with three messages is below.

[
    {
        "source": "APPNAME",
        "id": "1234567890",
        "recorded": "2018-06-07T00:44:22.584Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email1@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message1"
        }
    },
    {
        "source": "APPNAME",
        "id": "2345678901",
        "recorded": "2018-05-07T05:12:47.409Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email2@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "POLICY",
            "message": "Details:\nIP Address: 222.222.222.222\nCountry: AU\nNew Device: true \"Default Action\"\n"
        }
    },
    {
        "source": "APPNAME",
        "id": "3456789012",
        "recorded": "2018-05-07T05:12:58.137Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email3@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message3"
        }
    }   
]

I am not even going to bother showing what I have put in the props.conf file as it might be misleading.

I am facing this problem only when I am polling the JSON via REST API. In comparison, if I get the same JSON (flat file) and manually import it, Splunk is smart enough to break it down cleanly in individual events.

Any assistance would be highly appreciated. Thank you.

[claudio_palmeri · Accepted solution]

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

[pruthvikrishnap]

Hi,

This can be handled by creating a custome responsehandler i.e by editing responsehandler.py (/rest/bin/responsehandlers.py)

or you can exclude null values by adding below to your search.

search | where isnull(field name)

Let me know if this helps.

[claudio_palmeri]

Hi pruthvikrishnapolavarapu,
It does help, but I wouldn't know where to start to be honest. Is this something that you can guide me through or perhaps can you recommend a Knowledge Article to follow?

Should I use the responsehandler.py would I also need to modify props.conf at all? At the moment I have a combination of SEDCMD-remove_header, SEDCMD-remove_footer, LINE_BREAKER and TIME_PREFIX that are partially doing the job, but the REGEX need some fixing...

[claudio_palmeri]

I've seen this:

https://github.com/damiendallimore/SplunkModularInputsPythonFramework/blob/master/implementations/re...

and this

https://answers.splunk.com/answers/233620/how-to-use-custom-response-handlers-for-monitoring-1.html

but again, I wouldn't know where to start to configure them for my exact scenario...