topic Re: Routing and Filtering question in Getting Data In

Routing and Filtering question

yongly — Mon, 28 Sep 2020 12:21:17 GMT

Hi all,

I've come across a strange problem that I can't seem to figure out how to fix or troubleshoot. My problem is that for some reason, I can't seem to get my source or host recognised in the filter. I have a default discard_all rule that discards all logs sent to my filter server unless I define another stanza or rule to specifically handle those log files:

props.conf

[default]
TRANSFORMS-drop_all=discard_all

For some reason it ignores my source and host stanzas
[source::/var/log/nginx/access.log]
TRANSFORMS-ccp=allow_all

I have a filter set up with these entries in transforms.conf

[discard_all]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

Use this transform to allow and forward all entries from log file to indexer
[allow_all]
REGEX=.
DEST_KEY=queue
FORMAT=indexQueue

I know that when I change my default rule to allow_all. the file comes through to the indexer. I'm stumped because other log files seem to work fine.

Any ideas?

Re: Routing and Filtering question

kristian_kolb — Thu, 30 Aug 2012 06:38:00 GMT

Hm..why even monitor the files if you're going to discard most of them..?

Well, perhaps you have to specify both transforms on the same line, like;

[source::/var/log/nginx/access.log] 
TRANSFORMS-ccp = discard_all, allow_all

In this case it seems pretty silly, but perhaps you have more clever filters elsewhere.

Re: Routing and Filtering question

yongly — Thu, 30 Aug 2012 21:55:53 GMT

Well this is an intermediate server that we've been using for filtering. The idea is to keep control of what gets passed onto the indexer to avoid big files getting through and exceeding our licence. Hence a default discard and an explicit allow 🙂

I did wonder if another filter or stanza was picking it up and taking precedence but when I change the [default] to allow_all, the file comes through no problems.. this kind of suggests that for some reason it's not linking the access.log file and the stanza in props.conf.

I did try your suggestion anyway, but no luck. Any other ideas as to how I might troubleshoot this?

Re: Routing and Filtering question

kristian_kolb — Fri, 31 Aug 2012 06:59:08 GMT

did you remove the [default] discard transform?

Re: Routing and Filtering question

yongly — Fri, 31 Aug 2012 07:05:00 GMT

Yeh after some testing, I found that I had to remove it to get it to recognise the [source:..] stanza.

What I don't understand is why it worked with other sources and sourcetypes but not with this one?