Getting Data In
Highlighted

Rest API Modular Input: Exception performing request:

Explorer

I try to use the REST API to pull down data from Instagram. Everything seems to be ok, we can receive data, but only after a restart of Splunk or I click to the Save button in REST API UI and the configuration will be reloaded. In both situations all new connections triggered by the scheduler via rest.py is not working:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

The inputs.conf is looking like this:

[rest://instagramxxxxx]
auth
type = oauth2
backofftime = 10
endpoint = https://api.instagram.com/v1/users/xxxxx/media/recent
http
method = GET
httpproxy = http://xxxxx
https
proxy = http://xxxxx
index = instagram
indexerrorresponsecodes = 1
oauth2
accesstoken = xxxxx
oauth2
clientid = xxxxx
oauth2
clientsecret = xxxxx
polling
interval = 60
responsehandler = InstagramUserFeedEventHandler
response
type = json
streamingrequest = 0
url
args = minid=xxxxx,accesstoken=xxxxx,mintimestamp=0
sourcetype = _json
sequential
mode = 0
request_timeout = 30

I increased the logging level inside the rest.py to DEBUG and also for Splunk itself, but without any additional information that can help us.
I tried this with the restta 1.3.6 delivered within the instagram app from Splunkbase and version 1.4. Same behavior. We are running this on a Splunk 6.5.3 installed on a SUSE Linux Enterprise Server 12 SP2 (x8664).

Please, can someone help us with this issue?

Thanks!

Highlighted

Re: Rest API Modular Input: Exception performing request:

Splunk Employee
Splunk Employee

Hi

The app comes with the restta preconfigured. If you want to upgrade to the latest version of the restta (or just download manually from Splunkbase) you need to merge the files "rest-ta/bin/responsehandlers.py" which contains the custom Instagram responsehandler (InstagramUserFeedEventHandler) that I wrote for this input. Can you check to make sure that you have that piece of Instagram specific code I wrote in your app?

For generic troubleshooting of the rest TA I regularly use this search:

index=* rest sourcetype=splunkd component!=SearchParser| table _time component loglevel message

Let me know how you get along.

j

0 Karma
Highlighted

Re: Rest API Modular Input: Exception performing request:

Explorer

Hi Johan,

I followed the documentation and added all the specific code for Instagram to the current release of rest_ta(responsehandlers.py). But I got the same errors. Thats the output from the search:

 "2018-02-07T13:33:59.616+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:57.074+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:54.524+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:52.176+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx1"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx2"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436236.250000, max_age=1734051"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx3"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_xxx4"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436233.750000, max_age=1734046"
 "2018-02-07T13:32:58.799+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:56.306+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:53.803+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:51.270+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 1269 threads"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 21333 sockets"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.060+0100",ModularInputs,INFO,"Introspection setup completed for scheme ""rest""."
 "2018-02-07T13:32:45.856+0100",SpecFiles,INFO,"Found external scheme definition for stanza ""rest://"" with 36 parameters: endpoint, http_method, request_payload, auth_type, auth_user, auth_password, oauth1_client_key, oauth1_client_secret, oauth1_access_token, oauth1_access_token_secret, oauth2_token_type, oauth2_access_token, oauth2_refresh_token, oauth2_refresh_url, oauth2_refresh_props, oauth2_client_id, oauth2_client_secret, http_header_propertys, url_args, response_type, streaming_request, http_proxy, https_proxy, request_timeout, backoff_time, polling_interval, sequential_mode, sequential_stagger_time, index_error_response_codes, response_handler, response_handler_args, response_filter_pattern, custom_auth_handler, custom_auth_handler_args, delimiter, cookies"

Kind regards,
Christian

0 Karma
Highlighted

Re: Rest API Modular Input: Exception performing request:

Explorer

No one an idea what's wrong?

0 Karma
Highlighted

Re: Rest API Modular Input: Exception performing request:

Engager

Hello. Did you fix it? I have the same issue.

0 Karma
Highlighted

Re: Rest API Modular Input: Exception performing request:

Explorer

Hi, we stopped all activities within downloading instagram data, sorry.

0 Karma