topic Re: Rest API Modular Input: Exception performing request: in Getting Data In

Rest API Modular Input: Exception performing request:

swaro_ck — Tue, 29 Sep 2020 17:40:50 GMT

I try to use the REST API to pull down data from Instagram. Everything seems to be ok, we can receive data, but only after a restart of Splunk or I click to the Save button in REST API UI and the configuration will be reloaded. In both situations all new connections triggered by the scheduler via rest.py is not working:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

The inputs.conf is looking like this:

[rest://instagram_xxxxx]
auth_type = oauth2
backoff_time = 10
endpoint = https://api.instagram.com/v1/users/xxxxx/media/recent
http_method = GET
http_proxy = http://xxxxx
https_proxy = http://xxxxx
index = instagram
index_error_response_codes = 1
oauth2_access_token = xxxxx
oauth2_client_id = xxxxx
oauth2_client_secret = xxxxx
polling_interval = 60
response_handler = InstagramUserFeedEventHandler
response_type = json
streaming_request = 0
url_args = min_id=xxxxx,access_token=xxxxx,min_timestamp=0
sourcetype = _json
sequential_mode = 0
request_timeout = 30

I increased the logging level inside the rest.py to DEBUG and also for Splunk itself, but without any additional information that can help us.
I tried this with the rest_ta 1.3.6 delivered within the instagram app from Splunkbase and version 1.4. Same behavior. We are running this on a Splunk 6.5.3 installed on a SUSE Linux Enterprise Server 12 SP2 (x86_64).

Please, can someone help us with this issue?

Thanks!

Re: Rest API Modular Input: Exception performing request:

jbjerke_splunk — Tue, 29 Sep 2020 17:51:53 GMT

The app comes with the rest_ta preconfigured. If you want to upgrade to the latest version of the rest_ta (or just download manually from Splunkbase) you need to merge the files "rest-ta/bin/responsehandlers.py" which contains the custom Instagram responsehandler (InstagramUserFeedEventHandler) that I wrote for this input. Can you check to make sure that you have that piece of Instagram specific code I wrote in your app?

For generic troubleshooting of the rest TA I regularly use this search:

index=_* rest sourcetype=splunkd component!=SearchParser| table _time component log_level message

Let me know how you get along.

Re: Rest API Modular Input: Exception performing request:

swaro_ck — Wed, 07 Feb 2018 15:09:47 GMT

Hi Johan,

I followed the documentation and added all the specific code for Instagram to the current release of rest_ta(responsehandlers.py). But I got the same errors. Thats the output from the search:

 "2018-02-07T13:33:59.616+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:57.074+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:54.524+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:52.176+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx1"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx2"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436236.250000, max_age=1734051"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx3"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_xxx4"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436233.750000, max_age=1734046"
 "2018-02-07T13:32:58.799+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:56.306+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:53.803+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:51.270+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 1269 threads"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 21333 sockets"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.060+0100",ModularInputs,INFO,"Introspection setup completed for scheme ""rest""."
 "2018-02-07T13:32:45.856+0100",SpecFiles,INFO,"Found external scheme definition for stanza ""rest://"" with 36 parameters: endpoint, http_method, request_payload, auth_type, auth_user, auth_password, oauth1_client_key, oauth1_client_secret, oauth1_access_token, oauth1_access_token_secret, oauth2_token_type, oauth2_access_token, oauth2_refresh_token, oauth2_refresh_url, oauth2_refresh_props, oauth2_client_id, oauth2_client_secret, http_header_propertys, url_args, response_type, streaming_request, http_proxy, https_proxy, request_timeout, backoff_time, polling_interval, sequential_mode, sequential_stagger_time, index_error_response_codes, response_handler, response_handler_args, response_filter_pattern, custom_auth_handler, custom_auth_handler_args, delimiter, cookies"

Kind regards,
Christian

Re: Rest API Modular Input: Exception performing request:

swaro_ck — Thu, 08 Feb 2018 05:41:52 GMT

No one an idea what's wrong?

Re: Rest API Modular Input: Exception performing request:

mf_download — Wed, 10 Apr 2019 13:44:03 GMT

Hello. Did you fix it? I have the same issue.

Re: Rest API Modular Input: Exception performing request:

swaro_ck — Wed, 10 Apr 2019 13:48:35 GMT

Hi, we stopped all activities within downloading instagram data, sorry.