All Apps and Add-ons

Help with custom response handler for REST API Modular input.

quihong
Path Finder

Hello,

I've been tasked with ingesting some audit events from a online application (Lever Hire). I'm looking at using the REST API Modular input and need help parsing the data. Originally I thought I had to create a new sourcetype to do the parsing, but now I believe the correct method is to create a custom Response Handler.

I've found some examples here: https://github.com/damiendallimore/SplunkModularInputsPythonFramework/blob/master/implementations/re...

However, I'm not familiar enough with python to write my own response handler. I'm hoping someone in the community can quickly help me with the code.

Here is what the raw data looks like. Any help is appreciated.

{
    "data": [{
        "id": "5b628f1e-2bcf-45f7-90fa-7b1264987d42",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:added",
        "createdAt": 1515609233213,
        "target": {
            "type": "key",
            "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "w2jU",
                "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "85374119-2af3-48b2-838f-7821fb15ef7c",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:removed",
        "createdAt": 1515609175385,
        "target": {
            "type": "key",
            "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RhgT",
                "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "b368c76a-f7a5-4cc8-8201-ce4051847976",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515609122117,
        "target": {
            "type": "user",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "label": "Joe Mama"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
                "name": "Joe Mama",
                "email": "joe@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }
    }, {
        "id": "5b88b646-f141-4be7-a970-e39c56ce13ad",
        "user": {
            "role": "super admin",
            "id": "lever-support",
            "name": "Lever Support",
            "email": "support@lever.co"
        },
        "type": "key:added",
        "createdAt": 1515520786845,
        "target": {
            "type": "key",
            "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
            "label": "click-boarding"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RAof",
                "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
                "name": "click-boarding",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "c4ef90e5-449d-4a2b-a724-8cde900f1a1f",
        "user": {
            "role": "super admin",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "name": "superman",
            "email": "superman@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515456274871,
        "target": {
            "type": "user",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "label": "superman"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
                "name": "superman",
                "email": "superman@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }

    }],
    "hasNext": false
}
0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

0 Karma

quihong
Path Finder

Thank you very much!

Had to escape the quotes, other than that perfect.
[yoursourcetype]
TIME_PREFIX =\" createdAt\":

0 Karma

Damien_Dallimor
Ultra Champion

Please describe what it is you want the custom response handler to do with the raw JSON ?

0 Karma

quihong
Path Finder

Sorry I was not clear...

I would like the custom response handler to break out the raw json into individual events with proper timestamp (createdAt field). Each event starts with the {"id":

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...