topic Re: Splunk GUI Rest API Twitter Integration in All Apps and Add-ons

Splunk GUI Rest API Twitter Integration

aaronkorn — Fri, 11 Oct 2013 19:45:39 GMT

Hello,

We are looking use the Splunk REST gui to connect to Twitter and monitor feeds based on several URL parameters we care to search for.

We have the end point defined as https://api.twitter.com/1.1/search/tweets.json, have our authentication credentials entered, and for a sample URL Argument as q=UPMC to search twitter anything for UPMC returning in XML format. There is no data returning though but when I use this twitter dev app it works fine: https://dev.twitter.com/console. Anyone else having issue using the GUI Rest integration or have a better way to pull in twitter data based on keywords? Should we worry about defining Response Handler and other options in the config?

Thanks!

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Fri, 11 Oct 2013 23:35:06 GMT

Can you post your inputs.conf stanza for the Twitter Rest Input that is not working for you.

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Mon, 28 Sep 2020 14:57:35 GMT

Thanks for your response. Heres what we have so far:

[rest://Twitter]
auth_type = oauth1
endpoint = https://api.twitter.com/1.1/search/tweets.json
http_method = GET
index = twitter
index_error_response_codes = 1
oauth1_access_token = 1951974925-0Gmoi6JxxToMG4P7lEWX03xxxxxxxxxxxxxx
oauth1_access_token_secret = sYDyjNRz71Q0Wbbeni0RbuBoIQmUxxxxxxxxxxxxx
oauth1_client_key = vpIKhXBmLmqxxxxxxxxxxx
oauth1_client_secret = 0Vrp1WeP7g8NGewlTx2pMKcxxxxxxxxxxxxx
response_type = xml
sourcetype = twitter
streaming_request = 0
url_args = q=UPMC
polling_interval = 10
response_handler_args =

Re: Splunk GUI Rest API Twitter Integration

polifagbonanza — Sun, 13 Oct 2013 17:48:18 GMT

One of the issues may be that you have response_type set to xml, albeit you're pulling json data.

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Sun, 13 Oct 2013 18:21:15 GMT

Looks ok , I have the same setup in my twitter test stanza and it works..what search are you using ? try searching over "all time" for "index=twitter sourcetype=twitter"

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Mon, 14 Oct 2013 13:11:33 GMT

Thanks, I did that and it is only returning 1 event but when I search twitter.com/search for that query (UPMC) I get multiple results back. What would be the best way to leverage a real time stream for the keywords or only search for the most recent tweets?

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Wed, 16 Oct 2013 07:08:54 GMT

Well, you will be getting multiple events in the response document , but they are being indexed in Splunk as 1 single event. That is why the REST API Modular Input has Custom Response Handlers that you can plug in to parse the specific response you are getting back ie: split out the individual twitter events from the JSON response.
You add your custom response handler to bin/responsehandlers.py and declare it on the setup page for your REST Input Definition

Here is an example of what a custom handler might look like for the Twitter JSON response :

class TwitterEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        if response_type == "json":        
            output = json.loads(raw_response_output)
            last_tweet_indexed_id = 0
            for twitter_event in output["statuses"]:
                print_xml_stream(json.dumps(twitter_event))
                if "id_str" in twitter_event:
                    tweet_id = twitter_event["id_str"]
                    if tweet_id > last_tweet_indexed_id:
                        last_tweet_indexed_id = tweet_id

            if not "params" in req_args:
                req_args["params"] = {}

            req_args["params"]["since_id"] = last_tweet_indexed_id

        else:
            print_xml_stream(raw_response_output)

I see that the raw response back from twitter also has a created_at field for each event , which you can then use as your Splunk index time value.

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Sun, 20 Oct 2013 00:24:42 GMT

Thanks! This seems to be working pretty well now but we seem to be ingesting duplicate tweets every time it executes the API call. Is there anyway around this? I know in there are several different parameters to pass though in the request (can be found here: https://dev.twitter.com/docs/api/1.1/get/search/tweets). I imagine that we would want to use the since_id but how would you update this value in the call based off the last ingested event?

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Mon, 28 Sep 2020 15:01:18 GMT

Maintain a variable in responsehandlers.py that stores the last tweet id , and then use this as the since_id for your next request.And iteratively repeat this.

More advanced , but you could potentially also use the Splunk Python SDK from the response handler to execute a Splunk search and ask it for the latest tweet id that you indexed to use as your since_id.

Also , you can update your REST stanza "url_args" using the Python SDK , so if you needed to persist the since_id value back into you configuration (ie: to survive restarts), then you can do this also.

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Mon, 21 Oct 2013 19:57:27 GMT

Ok. Where should we start in the responsehandlers.py script? I guess I'm not fully understanding the purpose of the responsehandlers.py file and how we would go about passing this to the inputs.conf file.

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Tue, 22 Oct 2013 03:53:03 GMT

The REST Mod Input is generic , it can be used in an unknown number of scenarios , so I have to provide an extension mechanism for handling specific behaviours. This is the purpose of responsehandlers.py.So custom handling can also extend beyond output formatting to also being dynamically calculating URL arguments to add to the request.Such as the "since_id" argument which you need to calculate based on the latest tweet id that you processed.1 suggested 2 ways of performing this above , simple and more advanced.Updated above untested code snippet to show how this might potentially be done.

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Tue, 22 Oct 2013 19:44:43 GMT

Awesome! Just had to modify it slightly for the different id. Thanks for all your help

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Tue, 22 Oct 2013 19:45:06 GMT

id_str instead of id

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Tue, 22 Oct 2013 20:49:35 GMT

Nice. Updated the code example.

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Mon, 28 Sep 2020 15:03:01 GMT

So I thought this was working but it isnt. But it is close! Whenever we say "req_args["params"]["since_id"] = last_tweet_indexed_id" do we need to set this anywhere else or will it automatically be passed into the url parameter list? Or does since_id need to be added to the response handles arguments in the REST GUI?

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Wed, 23 Oct 2013 19:26:46 GMT

It is automatically passed into the url parameter list.
Trace back through the code in rest.py (the while loop at line 422) to see how this happens.

Re: Splunk GUI Rest API Twitter Integration

saad_siddiqi — Wed, 23 Oct 2013 22:31:37 GMT

I have the similar setting but getting below mentioned error in my splunkd.log, has anyone encountered this?

10-23-2013 11:30:00.231 +1300 ERROR ExecProcessor - message from "python /app/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: [Errno 8] _ssl.c:521: EOF occurred in violation of protocol

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Mon, 28 Sep 2020 15:03:29 GMT

Note : I just uploaded a new version of the REST Modular Input that now automatically persists any dynamically calculated URL arguments back into your inputs.conf stanzas (using the Splunk Python SDK).So if you restart the REST Modular Input stanza , it starts polling from where it last left off.I tested this all with the "TwitterEventHandler" (now included in version 1.3) and it worked perfectly for me.

Below is the intial stanza that I started with.

On each subsequent polling iteration, the url_args field gets dynamically updated with the latest tweet id as the since_id value.

[rest://twitter]
auth_type = oauth1
endpoint = https://api.twitter.com/1.1/search/tweets.json
http_method = GET
index = main
index_error_response_codes = 1
oauth1_access_token = 217362964-dtJVxxxxxxxxxUOY4Q0w
oauth1_access_token_secret = BWQ2LcQhxxxxxxxxxxxlf4o1B84mWrlE
oauth1_client_key = xYj5UxxxxxxxxxxxxOP97Q
oauth1_client_secret = LDdy4VoxxxxxxxxxxxxxxAI1HtlKU
polling_interval = 30
response_handler = TwitterEventHandler
response_type = json
sourcetype = rest_twitter
streaming_request = 0
url_args = q=music,since_id=0
disabled = 0

After the first polling, the since_id has now incremented :

[rest://twitter]
auth_type = oauth1
endpoint = https://api.twitter.com/1.1/search/tweets.json
http_method = GET
index = main
index_error_response_codes = 1
oauth1_access_token = 217362964-dtJVxxxxxxxxxUOY4Q0w
oauth1_access_token_secret = BWQ2LcQhxxxxxxxxxxxlf4o1B84mWrlE
oauth1_client_key = xYj5UxxxxxxxxxxxxOP97Q
oauth1_client_secret = LDdy4VoxxxxxxxxxxxxxxAI1HtlKU
polling_interval = 30
response_handler = TwitterEventHandler
response_type = json
sourcetype = rest_twitter
streaming_request = 0
url_args = q=music,since_id=393287846443753472
disabled = 0

Re: Splunk GUI Rest API Twitter Integration

aaronkorn — Thu, 24 Oct 2013 14:50:30 GMT

This is awesome! Thank you for the update. I notice that the since_id parameter is changing but we still seem to be ingesting duplicate tweets. Any idea what it could be?

Re: Splunk GUI Rest API Twitter Integration

Damien_Dallimor — Thu, 24 Oct 2013 18:55:15 GMT

I dont see this.What search are you using to determine this ?