All Apps and Add-ons

Help with custom response handler for REST API Modular input.

quihong
Path Finder

Hello,

I've been tasked with ingesting some audit events from a online application (Lever Hire). I'm looking at using the REST API Modular input and need help parsing the data. Originally I thought I had to create a new sourcetype to do the parsing, but now I believe the correct method is to create a custom Response Handler.

I've found some examples here: https://github.com/damiendallimore/SplunkModularInputsPythonFramework/blob/master/implementations/re...

However, I'm not familiar enough with python to write my own response handler. I'm hoping someone in the community can quickly help me with the code.

Here is what the raw data looks like. Any help is appreciated.

{
    "data": [{
        "id": "5b628f1e-2bcf-45f7-90fa-7b1264987d42",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:added",
        "createdAt": 1515609233213,
        "target": {
            "type": "key",
            "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "w2jU",
                "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "85374119-2af3-48b2-838f-7821fb15ef7c",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:removed",
        "createdAt": 1515609175385,
        "target": {
            "type": "key",
            "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RhgT",
                "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "b368c76a-f7a5-4cc8-8201-ce4051847976",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515609122117,
        "target": {
            "type": "user",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "label": "Joe Mama"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
                "name": "Joe Mama",
                "email": "joe@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }
    }, {
        "id": "5b88b646-f141-4be7-a970-e39c56ce13ad",
        "user": {
            "role": "super admin",
            "id": "lever-support",
            "name": "Lever Support",
            "email": "support@lever.co"
        },
        "type": "key:added",
        "createdAt": 1515520786845,
        "target": {
            "type": "key",
            "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
            "label": "click-boarding"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RAof",
                "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
                "name": "click-boarding",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "c4ef90e5-449d-4a2b-a724-8cde900f1a1f",
        "user": {
            "role": "super admin",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "name": "superman",
            "email": "superman@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515456274871,
        "target": {
            "type": "user",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "label": "superman"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
                "name": "superman",
                "email": "superman@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }

    }],
    "hasNext": false
}
0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

View solution in original post

0 Karma

quihong
Path Finder

Thank you very much!

Had to escape the quotes, other than that perfect.
[yoursourcetype]
TIME_PREFIX =\" createdAt\":

0 Karma

Damien_Dallimor
Ultra Champion

Please describe what it is you want the custom response handler to do with the raw JSON ?

0 Karma

quihong
Path Finder

Sorry I was not clear...

I would like the custom response handler to break out the raw json into individual events with proper timestamp (createdAt field). Each event starts with the {"id":

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!