All Apps and Add-ons

REST API Modular Input Add-on: Is it possible to parse a CSV dump from a REST URL?

tnerkar_splunk
Splunk Employee
Splunk Employee

It seems, indexed extractions don’t work with modular inputs. Is there another approach to parse a CSV dump from a REST URL into Splunk?

The data from the URL on the screen appears as:

"ServerName","Priority","VulnCount","IPAddress(ITSM)","Application(ITSM)","PrimaryBO(ITSM)","SecondaryBO(ITSM)","RebootWindow(ITSM)","StatusLabel(ITSM)","ScanResult(ADDM)","DescoveryEndTime(ADDM)","Uptime(Days)(ADDM)","Created(AD)","LastLogon(AD)","OperatingSystem(iPatch)","ReportingGroup(iPatch)","WeekOfMonth(iPatch)","LastBootTime(iPatch)","ePOStatus","PBStatus","SplunkStatus","TrendStatus"
"[n/a]","Other","0","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"[n/a]  what is this?","Other","0","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"aar-entbc-001","Other","0","Other","Blue Coat","DL-eBay-GET-Ops-Access-Management","DL-eBay-GET-Ops-Access-Management","Manual Reboot By BO","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"aar-entfs-001","Other","0","10.238.52.30","Filer","DL-eBay-GET-Ops-Storage","DL-eBay-GET-Ops-Hosting-all","Manual Reboot by ITS","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","Temp","Temp","None","Perm"
"aar-entfs-002","Other","0","10.238.52.31","Filer","DL-eBay-GET-Ops-Storage","DL-eBay-GET-Ops-Hosting-all","Manual Reboot by ITS","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","Temp","Temp","None",”Perm"

I pass URL arguments as: type=compr_all,format_type=text.

Thank you, appreciate your help.

-Tejal

0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

You will need to apply a custom response handler to split out the csv events.

You add this handler to rest_ta/bin/responsehandlers.py then wire it up in your stanza definition (Note : guiding example only based on my guesswork , so may need tweaking by you)

class RollOutCSVHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        import csv,io
        reader_list = csv.DictReader(io.StringIO(raw_response_output))
        for row in reader_list:     
            print_xml_stream(str(row))

alt text

View solution in original post

Damien_Dallimor
Ultra Champion

You will need to apply a custom response handler to split out the csv events.

You add this handler to rest_ta/bin/responsehandlers.py then wire it up in your stanza definition (Note : guiding example only based on my guesswork , so may need tweaking by you)

class RollOutCSVHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        import csv,io
        reader_list = csv.DictReader(io.StringIO(raw_response_output))
        for row in reader_list:     
            print_xml_stream(str(row))

alt text

tnerkar_splunk
Splunk Employee
Splunk Employee

Hi Damien,

I see errors in the logs:

09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" Exception in thread Thread-1:
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/lib/python2.7/threading.py", line 801, in __bootstrap_inner
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     self.run()
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/lib/python2.7/threading.py", line 754, in run
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     self.__target(*self.__args, **self.__kwargs)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/rest.py", line 521, in do_run
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     handle_output(r,r.text,response_type,req_args,endpoint)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/rest.py", line 614, in handle_output
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     RESPONSE_HANDLER_INSTANCE(response,output,type,req_args,endpoint)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 143, in __call__
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     print_xml_stream(row)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 304, in print_xml_stream
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     print "%s" % encodeXMLText(s)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 309, in encodeXMLText
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     text = text.replace("&", "&")
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" AttributeError: 'dict' object has no attribute 'replace'

Thanks,
Tejal

0 Karma

Damien_Dallimor
Ultra Champion

code sample updated.

please note , I'm only trying to guide you with guessed example code here , you can edit the code also based on your environment that I'm blind to.

0 Karma

tnerkar_splunk
Splunk Employee
Splunk Employee

Thanks Damien. I can work with the displayed data-set.

Regards,
Tejal Nerkar

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...