REST API Modular Input: index zip file error

[tamduong16]

I want to read a file in the zip archives and index it into Splunk with REST modular input app. The following is my code for the handler:

I have been doing research to get my code to work for a week now with no result. I tried running the code on an actual zip file and it works. But when I make change to the actual responsehandlers.py, it doesn't run the whole class and I believe it got stuck in the line file = zipfile.ZipFile(io.BytesIO(response_object.content)). Which is why I wrote the line print_xml_stream(response_type) and print_xml_stream("test2") to test it and only the first print works. The response_type that got index is "text", which lead me to wonder if rest modular input works for handling zip file since on the drop-down list, I only see json, xml and text. I read a few thing about python sdk that run on rest, would this be a good work-around?

[tamduong16 · Accepted solution]

So I figure out the answer to my question. The only thing I have to change in this code is the line

file = zipfile.ZipFile(io.BytesIO(response_object.content))

to

file = zipfile.ZipFile(StringIO.StringIO(response_object.content))

This problem is due to the difference between python 2.7 and 3.0.

[tamduong16]

HI Damien, Thanks for providing this app to us. There's no doubt this app is very useful. I used it to index my data in json format. In this code, I tried to run it without the for loop and if statement. But it won't index the string "test2" either. Which leads me to think it might be stuck at the line above it. My main concern is that it not grabbing the zip file. I don't have experience in dealing with response object from api so I don't know if the statement is right. Thanks for helping!

[Damien_Dallimor]

response_type is irrelevant for this use case , it is ignored.

This app is frequently used to index compressed/binary content over the years (I wrote it).

Your debugging logic is rather flawed however.

ie: perhaps you have an error in your regex pattern csv_file_to_index parameter that is not matching the file correctly in your zip, ergo , the if statement in the loop would not execute.