Getting Data In

How to parse a JSON from REST API?

claudio_palmeri
Explorer

Hi all,
I need some help parsing a JSON containing none/one/multiple nested messages that I have imported via REST API (poll). I am saying one or multiple or none as it depends on what the poll is retrieving from the REST API.

In the event that the poll is retrieving no new events, I would like Splunk not to show an empty entry (square brackets only) in the log. In the event of a single message, I'd like to remove the header/footer (square brackets) and display the event. In the event of multiple messages, I would like Splunk to remove header/footer as well as split the message in individual events, with associated timestamp.

I have been doing a lot of research, however I am struggling with the various attributes in $SPLUNK_HOME/etc/system/local/props.conf and with the related regular expressions to make the whole thing work.

A sample JSON with three messages is below.

[
    {
        "source": "APPNAME",
        "id": "1234567890",
        "recorded": "2018-06-07T00:44:22.584Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email1@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message1"
        }
    },
    {
        "source": "APPNAME",
        "id": "2345678901",
        "recorded": "2018-05-07T05:12:47.409Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email2@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "POLICY",
            "message": "Details:\nIP Address: 222.222.222.222\nCountry: AU\nNew Device: true \"Default Action\"\n"
        }
    },
    {
        "source": "APPNAME",
        "id": "3456789012",
        "recorded": "2018-05-07T05:12:58.137Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email3@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message3"
        }
    }   
]

I am not even going to bother showing what I have put in the props.conf file as it might be misleading.

I am facing this problem only when I am polling the JSON via REST API. In comparison, if I get the same JSON (flat file) and manually import it, Splunk is smart enough to break it down cleanly in individual events.

Any assistance would be highly appreciated. Thank you.

0 Karma
1 Solution

claudio_palmeri
Explorer

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

View solution in original post

0 Karma

claudio_palmeri
Explorer

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

0 Karma

pruthvikrishnap
Contributor

Hi,

This can be handled by creating a custome responsehandler i.e by editing responsehandler.py (/rest/bin/responsehandlers.py)

or you can exclude null values by adding below to your search.

search | where isnull(field name)

Let me know if this helps.

0 Karma

claudio_palmeri
Explorer

Hi pruthvikrishnapolavarapu,
It does help, but I wouldn't know where to start to be honest. Is this something that you can guide me through or perhaps can you recommend a Knowledge Article to follow?

Should I use the responsehandler.py would I also need to modify props.conf at all? At the moment I have a combination of SEDCMD-remove_header, SEDCMD-remove_footer, LINE_BREAKER and TIME_PREFIX that are partially doing the job, but the REGEX need some fixing...

0 Karma

claudio_palmeri
Explorer
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...