Hello guys
I have the following scenario:
[(?::){0}kube:*]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TRANSFORMS-set= setnull, allowEvents, dropEventsByText, dropEventsBySourcetype, set_sourcetype
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[allowEvents]
REGEX = LOG_INI|LOG_FINOK|LOG_FINEX|LOG_FINNEG
DEST_KEY = queue
FORMAT = indexQueue
[dropEventsBySourcetype]
SOURCE_KEY=MetaData:Sourcetype
REGEX = containerNumberOne|containerNumberTwo
DEST_KEY = queue
FORMAT = nullQueue
[dropEventsByText]
REGEX = debug|DEBUG
DEST_KEY = queue
FORMAT = nullQueue
[set_sourcetype]
SOURCE_KEY=MetaData:Sourcetype
REGEX = kube\:container\:(.*)\-re\-
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
These filters (and the sourcetype rename) have been working well for a while and as you may observe, they filter events based on a text contained in the log or by a text in the sourcetypes name.
The problem is that I have a new requirement. I need to drop events based on 2 rules at the same time: a sourcetype name and a text in the log. Specifically, there are some logs with the sourcetype name containerFour and the text LOG_INI that I need to drop. I guess I need something like this (but I know te conf is wrong):
[dropEventsBySourcetypeAndText]
SOURCE_KEY=MetaData:Sourcetype
REGEX = containerNumberFour
REGEX = BCI_INI
DEST_KEY = queue
FORMAT = nullQueue
Does someone know what i need to do?
Thanks in advance
Finally I resolved my problem filtering by "source". The structure for containers is something like this: /var/log/containers/*log
So, my stanza now is:
[source::/var/log/containers/(containerNumberFour*|containerNumberFive*)]
Thanks @scelikok for your help
Finally I resolved my problem filtering by "source". The structure for containers is something like this: /var/log/containers/*log
So, my stanza now is:
[source::/var/log/containers/(containerNumberFour*|containerNumberFive*)]
Thanks @scelikok for your help
Glad you got it working!
You can also optimize by moving those filters to the collector. If you don't want the data, it can be filtered at the inputs just like a UF!! By default we pick up all the things...
Best way to get rid of logs is to never pick them up!!!
You're welcome @hernanrodriguez,
Since it is regex below should work;
[(?::){0}kube:containerNumberFour.*]
TRANSFORMS-set= drop_BCI_INI_EventsByText
If this reply helps you an upvote is appreciated.
It didn't work 😞
Actually one of the sourcetypes is a bit different:
[(?::){0}kube:container:ms\-loan\-mobile\-exp.*]
Could be a problem with the special characters: "-"?
something like this should work?
[.*ms\-loan\-mobile\-exp.*]
It seems, it may work. It is better to test your regex with www.regex101.com .
Hi @hernanrodriguez,
You can use separate props stanza using specific sourcetype and transforms for this requirement like below;
props.conf
[(?::){0}kube:containerNumberFour]
TRANSFORMS-set= drop_BCI_INI_EventsByText
transforms.conf
[drop_BCI_INI_EventsByText]
REGEX = BCI_INI
DEST_KEY = queue
FORMAT = nullQueue
If this reply helps you an upvote is appreciated.
I tried that yesterday, but It wasn't work.
The sourcetype is dynamic, so I configured something like this:
[(?::){0}kube:containerNumberFour*]
TRANSFORMS-set= drop_BCI_INI_EventsByText
Can I use the * symbol to match any sourcetype with the prefix "containerNumberFour"?
I mean, my sourcetypes are something like:
kube:containerNumberFour-rev-1.1
kube:containerNumberFour-rev-1.2
kube:containerNumberFour-rev-2.0
Thanks @scelikok for your help