Getting Data In

How to parse a JSON from REST API?

Hi all,
I need some help parsing a JSON containing none/one/multiple nested messages that I have imported via REST API (poll). I am saying one or multiple or none as it depends on what the poll is retrieving from the REST API.

In the event that the poll is retrieving no new events, I would like Splunk not to show an empty entry (square brackets only) in the log. In the event of a single message, I'd like to remove the header/footer (square brackets) and display the event. In the event of multiple messages, I would like Splunk to remove header/footer as well as split the message in individual events, with associated timestamp.

I have been doing a lot of research, however I am struggling with the various attributes in $SPLUNK_HOME/etc/system/local/props.conf and with the related regular expressions to make the whole thing work.

A sample JSON with three messages is below.

[
    {
        "source": "APPNAME",
        "id": "1234567890",
        "recorded": "2018-06-07T00:44:22.584Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email1@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message1"
        }
    },
    {
        "source": "APPNAME",
        "id": "2345678901",
        "recorded": "2018-05-07T05:12:47.409Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email2@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "POLICY",
            "message": "Details:\nIP Address: 222.222.222.222\nCountry: AU\nNew Device: true \"Default Action\"\n"
        }
    },
    {
        "source": "APPNAME",
        "id": "3456789012",
        "recorded": "2018-05-07T05:12:58.137Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email3@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message3"
        }
    }   
]

I am not even going to bother showing what I have put in the props.conf file as it might be misleading.

I am facing this problem only when I am polling the JSON via REST API. In comparison, if I get the same JSON (flat file) and manually import it, Splunk is smart enough to break it down cleanly in individual events.

Any assistance would be highly appreciated. Thank you.

0 Karma
1 Solution

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

View solution in original post

0 Karma

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

View solution in original post

0 Karma

Contributor

Hi,

This can be handled by creating a custome responsehandler i.e by editing responsehandler.py (/rest/bin/responsehandlers.py)

or you can exclude null values by adding below to your search.

search | where isnull(field name)

Let me know if this helps.

0 Karma

Hi pruthvikrishnapolavarapu,
It does help, but I wouldn't know where to start to be honest. Is this something that you can guide me through or perhaps can you recommend a Knowledge Article to follow?

Should I use the responsehandler.py would I also need to modify props.conf at all? At the moment I have a combination of SEDCMD-remove_header, SEDCMD-remove_footer, LINE_BREAKER and TIME_PREFIX that are partially doing the job, but the REGEX need some fixing...

0 Karma

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!