Getting Data In

Rest API Modular Input: Exception performing request:

swaro_ck
Path Finder

I try to use the REST API to pull down data from Instagram. Everything seems to be ok, we can receive data, but only after a restart of Splunk or I click to the Save button in REST API UI and the configuration will be reloaded. In both situations all new connections triggered by the scheduler via rest.py is not working:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

The inputs.conf is looking like this:

[rest://instagram_xxxxx]
auth_type = oauth2
backoff_time = 10
endpoint = https://api.instagram.com/v1/users/xxxxx/media/recent
http_method = GET
http_proxy = http://xxxxx
https_proxy = http://xxxxx
index = instagram
index_error_response_codes = 1
oauth2_access_token = xxxxx
oauth2_client_id = xxxxx
oauth2_client_secret = xxxxx
polling_interval = 60
response_handler = InstagramUserFeedEventHandler
response_type = json
streaming_request = 0
url_args = min_id=xxxxx,access_token=xxxxx,min_timestamp=0
sourcetype = _json
sequential_mode = 0
request_timeout = 30

I increased the logging level inside the rest.py to DEBUG and also for Splunk itself, but without any additional information that can help us.
I tried this with the rest_ta 1.3.6 delivered within the instagram app from Splunkbase and version 1.4. Same behavior. We are running this on a Splunk 6.5.3 installed on a SUSE Linux Enterprise Server 12 SP2 (x86_64).

Please, can someone help us with this issue?

Thanks!

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi

The app comes with the rest_ta preconfigured. If you want to upgrade to the latest version of the rest_ta (or just download manually from Splunkbase) you need to merge the files "rest-ta/bin/responsehandlers.py" which contains the custom Instagram responsehandler (InstagramUserFeedEventHandler) that I wrote for this input. Can you check to make sure that you have that piece of Instagram specific code I wrote in your app?

For generic troubleshooting of the rest TA I regularly use this search:

index=_* rest sourcetype=splunkd component!=SearchParser| table _time component log_level message

Let me know how you get along.

j

0 Karma

swaro_ck
Path Finder

Hi Johan,

I followed the documentation and added all the specific code for Instagram to the current release of rest_ta(responsehandlers.py). But I got the same errors. Thats the output from the search:

 "2018-02-07T13:33:59.616+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:57.074+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:54.524+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:52.176+0100",ExecProcessor,ERROR,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Exception performing request:"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx1"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx2"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436236.250000, max_age=1734051"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_tags_xxx3"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=0.000000, max_age=0"
 "2018-02-07T13:33:16.676+0100",Metrics,INFO,"group=per_source_thruput, series=""rest://instagram_xxx4"", kbps=2.293122, eps=1.307874, kb=70.132812, ev=40, avg_age=436233.750000, max_age=1734046"
 "2018-02-07T13:32:58.799+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:56.306+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:53.803+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:51.270+0100",ExecProcessor,INFO,"message from ""python /opt/splunk/etc/apps/rest_ta/bin/rest.py"" Starting new HTTPS connection (1): api.instagram.com"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 1269 threads"
 "2018-02-07T13:32:46.306+0100",loader,INFO,"Limiting REST HTTP server to 21333 sockets"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.121+0100",ExecProcessor,INFO,"New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py"
 "2018-02-07T13:32:46.060+0100",ModularInputs,INFO,"Introspection setup completed for scheme ""rest""."
 "2018-02-07T13:32:45.856+0100",SpecFiles,INFO,"Found external scheme definition for stanza ""rest://"" with 36 parameters: endpoint, http_method, request_payload, auth_type, auth_user, auth_password, oauth1_client_key, oauth1_client_secret, oauth1_access_token, oauth1_access_token_secret, oauth2_token_type, oauth2_access_token, oauth2_refresh_token, oauth2_refresh_url, oauth2_refresh_props, oauth2_client_id, oauth2_client_secret, http_header_propertys, url_args, response_type, streaming_request, http_proxy, https_proxy, request_timeout, backoff_time, polling_interval, sequential_mode, sequential_stagger_time, index_error_response_codes, response_handler, response_handler_args, response_filter_pattern, custom_auth_handler, custom_auth_handler_args, delimiter, cookies"

Kind regards,
Christian

0 Karma

swaro_ck
Path Finder

No one an idea what's wrong?

0 Karma

mf_download
Engager

Hello. Did you fix it? I have the same issue.

0 Karma

swaro_ck
Path Finder

Hi, we stopped all activities within downloading instagram data, sorry.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...