How to import data to Splunk via HTTP GET request?

[tamduong16]

I've been looking for a way to import contents from an http get request with Splunk without success. At first, I thought I could do this by using Rest Api section that build-in Splunk. But after I give it a url to do an http get request, my search return no event. I thought this is all I have to do to get content from the page to Splunk. The documentation for this section is very confusing and I don't know where to start. At this point, I don't know if Rest Api is the answer to my question. Does anyone know of a way I could get content with an http get request in Splunk?

[sjalexander · Accepted solution]

Did your search return anything at all? Normally, with the API you run a search and the query returns a search SID, then you use the SID to query again for the results of the search.

You would usually build your search as a set of parameters and pass it to an endpoint, and get a SID back from that. Here's an example snippet from some python code I wrote to get bucket info from a dbinspect search:

 params = "search=%7Cdbinspect%20index%3D%2A%20latest%3Dnow%20earliest%3D-99y&exec_mode=blocking"
 url = "https://" + searchhead + ":8089/services/search/jobs/"
 item = json.loads(fetchdata(url,params))
 try:
     sid = str(item['sid'])
 except:
     sys.exit(1)
 url = "https://" + searchhead + ":8089/services/search/jobs/" + sid + "/results/"
 urldata = fetchdata(url)

You can see the query is in the params variable, and I pass it to the search head (the url variable) as a query parameter (the fetchdata call). the result is returned to the item[] array which I grab the SID from (the try block: sid = str item['sid']).
Once I have the SID I build a new url var with the SID (second from last line in the example) then call it to return the result data.

[Sukisen1981]

in addition you can use workflow actions and alerts to GET / POST http actions as well. Remember, the REST api will poll the GET url continuously at the polling interval. Suggest read up workflow and alert actions as well, but what @sjalexander is asking is most important - if your rest api integration is not working and not fetching a SID then it could be that you have missed some configuration, are you sure you have passed the tokens / keys needed to fetch data from your GET url? Mostly all api urls will have some authentication mechanism....