How do I configure Splunk to filter in only Tweets that I want to index?

[BlueSocket]

Dear All,

I am a n00b and working with the twitter2 App to do some analysis of responses to a TV programme, for instance, TopGear. I COULD take all of the Tweets that are generated in the world and then search them all for just #TopGear, but this is very wasteful and requires much more data to be ingested than will actually be used at all in the system (not to mention use up much index licensing).

So, I need to use props.conf and transforms.conf to filter in only the Tweets that I am looking for. This sounds like something that should be easy, as there are a few entries in the Splunk manuals that itemize this process, but I cannot see that any filtering is happening. Is there a log that I can switch on that will show me what filtering is happening, so that I can work out the problem?

Also, here is the local props.conf:

[script:/opt/splunk/etc/apps/twitter2/bin/stream_tweets.py]
TRANSFORMS-set= setnull,setparsing

And here is the contents of the transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = /#(?i:topgear)\b/
DEST_KEY = queue
FORMAT = indexQueue

I am not sure where I am going wrong. I am just not seeing any filtering at all, where I think that I should be!

Thanks in advance,

BlueSocket

[woodcock · Accepted solution]

Your stanza head in 'props.conf is wrong; change [script:/opt/splunk/etc/apps/twitter2/bin/stream_tweets.py], to [myTweetSourcetype] instead.

[BlueSocket]

Sorry, do you mean to change it to be:

[twitter]
TRANSFORMS-set= setnull,setparsing

Yes, I think that this works... (I have other issues now)!