Twitter Connection with REST API Issues

[theouhuios]

Hello

I am trying to get a stream of twitter data for a string to analyze some results. However I have not been able to do that successfully. Here are the two inputs which I tried and issues with each one of them.

[rest://resttwitter]
auth_type = oauth1
endpoint = https://stream.twitter.com/1.1/statuses/filter.json
http_method = GET
index = twitter
index_error_response_codes = 0
oauth1_access_token = 159023053-ZHaAF0KIVFoCfDvQvCvniw3oU3R4s3txxxxxx
oauth1_access_token_secret = E6AeYZTQnPk8s7C2UN4whxE1rKRozRugrvRdxxxxx
oauth1_client_key = VRyu5GvIPkhbC71xxxxx
oauth1_client_secret = F3THGsofgsV7xifw5FS78YMQ1t4jzderxxxxxxx
response_type = json
sourcetype = rest_twitter1
streaming_request = 1
url_args = track=twitter

With this endpoint I always get the error that its 401. I have confirmed the token again and again and still dont understand on why this error happens.

When I try this input with a different endpoint

[rest://resttwitter]
    auth_type = oauth1
    endpoint = https://api.twitter.com/1.1/search/tweets.json
    http_method = GET
    index = twitter
    index_error_response_codes = 0
    oauth1_access_token = 159023053-ZHaAF0KIVFoCfDvQvCvniw3oU3R4s3txxxxxx
    oauth1_access_token_secret = E6AeYZTQnPk8s7C2UN4whxE1rKRozRugrvRdxxxxx
    oauth1_client_key = VRyu5GvIPkhbC71xxxxx
    oauth1_client_secret = F3THGsofgsV7xifw5FS78YMQ1t4jzderxxxxxxx
    response_type = json
    sourcetype = rest_twitter1
    streaming_request = 1
    url_args = q=PSEC

I get duplicate events generated. Also these events are not exactly breaking up as json events.

Here is the props.conf.

[rest_twitter1]
TIME_PREFIX = "created_at":"
TIME_FORMAT = %a %b %d %H:%M:%S %z %Y
MAX_TIMESTAMP_LOOKAHEAD = 40

Any help please?

[Damien_Dallimor]

Regarding question 1 on the streaming connection :

I tried it with a POST and GET HTTP request and it worked.

Check your auth tokens , is there a hidden whitespace character perhaps ?

Regarding question 2 :

The REST API Modular Input is generic for use against any REST API.

In order to apply specific behavior to a particular REST source , you can declare a custom response handler.

It just so happens that the current REST API Modular Input comes with an example handler for Twitter , so you need to declare this in your setup. This will break up the tweets in the JSON response into individual events , and also keep track of where you are in the tweet stream using the "since_id" argument and persist this back to Splunk to survive restarts.

Note this handler is just for the polling API , not the streaming API.