REST API Modular Input Add-on: Is it possible to parse a CSV dump from a REST URL?

[tnerkar_splunk]

It seems, indexed extractions don’t work with modular inputs. Is there another approach to parse a CSV dump from a REST URL into Splunk?

The data from the URL on the screen appears as:

"ServerName","Priority","VulnCount","IPAddress(ITSM)","Application(ITSM)","PrimaryBO(ITSM)","SecondaryBO(ITSM)","RebootWindow(ITSM)","StatusLabel(ITSM)","ScanResult(ADDM)","DescoveryEndTime(ADDM)","Uptime(Days)(ADDM)","Created(AD)","LastLogon(AD)","OperatingSystem(iPatch)","ReportingGroup(iPatch)","WeekOfMonth(iPatch)","LastBootTime(iPatch)","ePOStatus","PBStatus","SplunkStatus","TrendStatus"
"[n/a]","Other","0","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"[n/a]  what is this?","Other","0","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"aar-entbc-001","Other","0","Other","Blue Coat","DL-eBay-GET-Ops-Access-Management","DL-eBay-GET-Ops-Access-Management","Manual Reboot By BO","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","None","None","None","None"
"aar-entfs-001","Other","0","10.238.52.30","Filer","DL-eBay-GET-Ops-Storage","DL-eBay-GET-Ops-Hosting-all","Manual Reboot by ITS","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","Temp","Temp","None","Perm"
"aar-entfs-002","Other","0","10.238.52.31","Filer","DL-eBay-GET-Ops-Storage","DL-eBay-GET-Ops-Hosting-all","Manual Reboot by ITS","Deployed","Other","Other","Other","Other","Other","Other","Other","Other","Other","Temp","Temp","None",”Perm"

I pass URL arguments as: type=compr_all,format_type=text.

Thank you, appreciate your help.

-Tejal

[Damien_Dallimor · Accepted solution]

You will need to apply a custom response handler to split out the csv events.

You add this handler to rest_ta/bin/responsehandlers.py then wire it up in your stanza definition (Note : guiding example only based on my guesswork , so may need tweaking by you)

class RollOutCSVHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        import csv,io
        reader_list = csv.DictReader(io.StringIO(raw_response_output))
        for row in reader_list:     
            print_xml_stream(str(row))

[tnerkar_splunk]

Hi Damien,

I see errors in the logs:

09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" Exception in thread Thread-1:
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/lib/python2.7/threading.py", line 801, in __bootstrap_inner
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     self.run()
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/lib/python2.7/threading.py", line 754, in run
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     self.__target(*self.__args, **self.__kwargs)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/rest.py", line 521, in do_run
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     handle_output(r,r.text,response_type,req_args,endpoint)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/rest.py", line 614, in handle_output
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     RESPONSE_HANDLER_INSTANCE(response,output,type,req_args,endpoint)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 143, in __call__
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     print_xml_stream(row)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 304, in print_xml_stream
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     print "%s" % encodeXMLText(s)
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"   File "/Applications/Splunk/etc/apps/rest_ta/bin/responsehandlers.py", line 309, in encodeXMLText
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py"     text = text.replace("&", "&")
09-14-2016 14:33:06.035 -0700 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" AttributeError: 'dict' object has no attribute 'replace'

Thanks,
Tejal

[Damien_Dallimor]

code sample updated.

please note , I'm only trying to guide you with guessed example code here , you can edit the code also based on your environment that I'm blind to.

[tnerkar_splunk]

Thanks Damien. I can work with the displayed data-set.

Regards,
Tejal Nerkar