All Apps and Add-ons

How to obtain data through the FreshDesk API in REST API Modular Input?

gonzaloAgui
Explorer

Hi,
I need to obtain data from FreshDesk through the FreshDesk API using a url param named "updated_since" (this is a datetime string with format "%Y-%m-%dT%H:%M:%SZ") like a checkpoint from the last ticket indexed. And adding 1 second to the checkpoint because otherwise the last ticket is doubled.
The problem is (I THINK) that the URL Arguments passed from the Response Handler to the rest.py change the format to this:

URL Arguments:
updated_since=1970-10-10 10:10:10

And this generate a Bad request to the API:
03-26-2018 16:27:32.652 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 400 Client Error: Bad Request

My response handler script:

class FreshDeskResponseHandler:
    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        if response_type == "json":
            output = json.loads(raw_response_output)
            datetime_format = '%Y-%m-%d %H:%M:%S'
            last_update_at = "1970-10-10 10:10:10"
            for freshdesk_event in output:
                print_xml_stream(json.dumps(freshdesk_event))
                if "updated_at" in freshdesk_event:
                    update_at_date = (freshdesk_event["updated_at"].replace("T", " "))[:-1]
                    if update_at_date > last_update_at:
                        update_one_sec = str(datetime.datetime.strptime(update_at_date, datetime_format) + datetime.timedelta(seconds=1))
                        update_one_sec = (update_one_sec.replace(" ", "T")) + str('Z')
                        last_update_at = update_one_sec
            if not "params" in req_args:
                req_args["params"] = {}

            req_args["params"]["updated_since"] = str(last_update_at.replace(" ", "T") + str('Z'))

        else:
            print_xml_stream(raw_response_output)

My inputs .conf:

[rest://TEST - input freshdeck]
auth_password = PASSWORD
auth_type = basic
auth_user = USERNAME
endpoint = https://DOMAIN.freshdesk.com/api/v2/tickets
http_method = GET
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = json_freshdesk
streaming_request = 0
cookies = _x_w=24_5
disabled = 0
host = fresh_desk_api
index = testfresh
polling_interval = 60
response_handler_args =
response_handler = FreshDeskResponseHandler
url_args = updated_since=1970-10-10 10:10:10

Has someone encountered a similar problem or knows a workaround to this?
Thanks in advance!

0 Karma

Damien_Dallimor
Ultra Champion

Are you able to get a trace of the actual HTTP GET request sent out , maybe using wireshark or something similar to capture it.

0 Karma

gonzaloAgui
Explorer

Hi Damien, thank you for your quick response. I am trying to trace the http request but failing in the attempt. Could you guide me please? the instance of splunk is in a virtual machine Centos without graphical interface and I do not have access to the server that hosts it.
I have tried with tcpdump, netstat, lsof, iptraf without success.

0 Karma

Damien_Dallimor
Ultra Champion

Try using Splunk Stream 🙂 https://splunkbase.splunk.com/app/1809/

0 Karma

gonzaloAgui
Explorer

I'm already configure splunk stream, but I still can't see the URL with the parameters or the body of the package.
I see data from this sources:

stream:Splunk_DNSIntegrity
stream:Splunk_DNSRequestResponse

stream:Splunk_SSLActivity

stream:Splunk_HTTPURI

stream:http

Is there something I'm missing? let me know if you need a sample of the events.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...