Response Filter for REST API Modular Input not filtering

[efullman]

I'm using one of the public Twitter streaming APIs. I was hoping I could cause it to only index events that match a Regex in the Response Filter (as advertised), but it seems to ignore it. I tested the Regex, but I'm missing some detailed documentation that might answer my questions.

Has anyone gotten this to work? I was just searching for this - lang":"en"

I'm assuming this would reject all events that don't include this, so as a result, only English tweets would flow through to the index. Unfortunately, everything still flows through.

[Damien_Dallimor]

What is the Python Regex pattern that you are entering in your setup for the response filter parameter ? Perhaps this pattern is wrong.

[efullman]

Damien,

Thanks for the response. I think I understand the purpose of this function now.

It doesn't filter the events that are indexed, it filters what is indexed in each event?

So you need capture groups (...) in the REGEX, and the captures get indexed?

Or is there a way to disregard a whole event? Such as nesting the captures (...(...)...) where one of the inner captures matches to a literal like "lang:"en and when the REST API modular input can't find the literal it ejects the whole event.?

Thanks for your thoughts.

[Damien_Dallimor]

I think you are confusing yourself.

It is rather simple.

Write a REGEX. If the REGEX matches the raw response, it gets processed and sent to Splunk for indexing. Else , it gets discarded.

[efullman]

Just searching for the literal - lang":"en"to start. It should match at least 40% of the time.

[Damien_Dallimor]

Thats doesn't look like a Python Regex Pattern that would match an entire response.

I would suggest getting some data of an entire raw response and trying out some Regexes first.
There are some online tools for trying out Regexes such as http://pythex.org/