Hello
I am trying to get a stream of twitter data for a string to analyze some results. However I have not been able to do that successfully. Here are the two inputs which I tried and issues with each one of them.
[rest://resttwitter]
auth_type = oauth1
endpoint = https://stream.twitter.com/1.1/statuses/filter.json
http_method = GET
index = twitter
index_error_response_codes = 0
oauth1_access_token = 159023053-ZHaAF0KIVFoCfDvQvCvniw3oU3R4s3txxxxxx
oauth1_access_token_secret = E6AeYZTQnPk8s7C2UN4whxE1rKRozRugrvRdxxxxx
oauth1_client_key = VRyu5GvIPkhbC71xxxxx
oauth1_client_secret = F3THGsofgsV7xifw5FS78YMQ1t4jzderxxxxxxx
response_type = json
sourcetype = rest_twitter1
streaming_request = 1
url_args = track=twitter
With this endpoint I always get the error that its 401. I have confirmed the token again and again and still dont understand on why this error happens.
When I try this input with a different endpoint
[rest://resttwitter]
auth_type = oauth1
endpoint = https://api.twitter.com/1.1/search/tweets.json
http_method = GET
index = twitter
index_error_response_codes = 0
oauth1_access_token = 159023053-ZHaAF0KIVFoCfDvQvCvniw3oU3R4s3txxxxxx
oauth1_access_token_secret = E6AeYZTQnPk8s7C2UN4whxE1rKRozRugrvRdxxxxx
oauth1_client_key = VRyu5GvIPkhbC71xxxxx
oauth1_client_secret = F3THGsofgsV7xifw5FS78YMQ1t4jzderxxxxxxx
response_type = json
sourcetype = rest_twitter1
streaming_request = 1
url_args = q=PSEC
I get duplicate events generated. Also these events are not exactly breaking up as json events.
Here is the props.conf.
[rest_twitter1]
TIME_PREFIX = "created_at":"
TIME_FORMAT = %a %b %d %H:%M:%S %z %Y
MAX_TIMESTAMP_LOOKAHEAD = 40
Any help please?
Regarding question 1 on the streaming connection :
I tried it with a POST and GET HTTP request and it worked.
Check your auth tokens , is there a hidden whitespace character perhaps ?
Regarding question 2 :
The REST API Modular Input is generic for use against any REST API.
In order to apply specific behavior to a particular REST source , you can declare a custom response handler.
It just so happens that the current REST API Modular Input comes with an example handler for Twitter , so you need to declare this in your setup. This will break up the tweets in the JSON response into individual events , and also keep track of where you are in the tweet stream using the "since_id" argument and persist this back to Splunk to survive restarts.
Note this handler is just for the polling API , not the streaming API.