All Apps and Add-ons

Help with custom response handler for REST API Modular input.

quihong
Path Finder

Hello,

I've been tasked with ingesting some audit events from a online application (Lever Hire). I'm looking at using the REST API Modular input and need help parsing the data. Originally I thought I had to create a new sourcetype to do the parsing, but now I believe the correct method is to create a custom Response Handler.

I've found some examples here: https://github.com/damiendallimore/SplunkModularInputsPythonFramework/blob/master/implementations/re...

However, I'm not familiar enough with python to write my own response handler. I'm hoping someone in the community can quickly help me with the code.

Here is what the raw data looks like. Any help is appreciated.

{
    "data": [{
        "id": "5b628f1e-2bcf-45f7-90fa-7b1264987d42",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:added",
        "createdAt": 1515609233213,
        "target": {
            "type": "key",
            "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "w2jU",
                "id": "8d0501a2-f613-4154-b2e7-fc4b416ad213",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "85374119-2af3-48b2-838f-7821fb15ef7c",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "key:removed",
        "createdAt": 1515609175385,
        "target": {
            "type": "key",
            "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
            "label": "Splunk"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RhgT",
                "id": "21b0fb88-006d-4a9a-a1e1-2164fcd8d243",
                "name": "Splunk",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "b368c76a-f7a5-4cc8-8201-ce4051847976",
        "user": {
            "role": "super admin",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "name": "Joe Mama",
            "email": "joe@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515609122117,
        "target": {
            "type": "user",
            "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
            "label": "Joe Mama"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "8810816c-03da-48db-b3c1-d47a8f5c024f",
                "name": "Joe Mama",
                "email": "joe@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }
    }, {
        "id": "5b88b646-f141-4be7-a970-e39c56ce13ad",
        "user": {
            "role": "super admin",
            "id": "lever-support",
            "name": "Lever Support",
            "email": "support@lever.co"
        },
        "type": "key:added",
        "createdAt": 1515520786845,
        "target": {
            "type": "key",
            "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
            "label": "click-boarding"
        },
        "meta": {
            "key": {
                "tokenLastFour": "RAof",
                "id": "82cedc33-87ff-4d68-bc44-7dcc7559da4c",
                "name": "click-boarding",
                "partner": false,
                "service": "data-api"
            }
        }
    }, {
        "id": "c4ef90e5-449d-4a2b-a724-8cde900f1a1f",
        "user": {
            "role": "super admin",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "name": "superman",
            "email": "superman@mama.org"
        },
        "type": "user.authentication:succeeded",
        "createdAt": 1515456274871,
        "target": {
            "type": "user",
            "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
            "label": "superman"
        },
        "meta": {
            "user": {
                "role": "super admin",
                "id": "cd6751d7-998a-451b-ab22-fb2e0fa96da5",
                "name": "superman",
                "email": "superman@mama.org"
            },
            "authentication": {
                "method": "direct"
            }
        }

    }],
    "hasNext": false
}
0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

Handler example below.

Then use standard Splunk timestamp extraction in props.conf for your sourcetype to use the createdAt time as the prefix.

[yoursourcetype]
TIME_PREFIX = createdAt": 

Handler

class ExampleHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for item in output["data"]:
                print_xml_stream(json.dumps(item))   
        else:
            print_xml_stream(raw_response_output)

alt text

0 Karma

quihong
Path Finder

Thank you very much!

Had to escape the quotes, other than that perfect.
[yoursourcetype]
TIME_PREFIX =\" createdAt\":

0 Karma

Damien_Dallimor
Ultra Champion

Please describe what it is you want the custom response handler to do with the raw JSON ?

0 Karma

quihong
Path Finder

Sorry I was not clear...

I would like the custom response handler to break out the raw json into individual events with proper timestamp (createdAt field). Each event starts with the {"id":

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...