forwarder with regex

[simo]

Hi All,

my solution foresees a heavy forwarder that sends data to an indexer, in the transforms.conf file I have a regex, which allows me to filter through string only the lines I need

regex example
REGEX = ^.*(?:SIMONE|MARCO).*

file.log example to monitor
xxxxx|xxxxx|xxxxx|SIMONE|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|VALERIO|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|SILVIA|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|MARCO|xxxxx|xxxxx|xxxxx

I am acknowledging these errors

ERROR Regex - Failed in pcre_exec: Error PCRE_ERROR_MATCHLIMIT for regex:
WARN regexExtractionProcessor - Regular expression for stanza xxxxx exceeded configured PCRE match limit. One or more fields might not have their values extracted, which can lead to incorrect search results. Fix the regular expression to improve search performance or increase the MATCH_LIMIT in props.conf to include the missing field extractions.

I wanted to know if there is a different way to filter the data to send without regex?

Best Regards,
Simone

[ITWhisperer]

Try your regex without the greedy .* since this is implied i.e. the expression will be matched anywhere in the event

REGEX = (?:SIMONE|MARCO)

[simo]

Hi @ITWhisperer,

Thanks for the reply, I would like to remove the regex to filter. it's possible?

in the regex it is also present

REGEX = (?:SIMONE|MARCO|SIMONE.*HOME)

xxxxx|xxxxx|SIMONE|xxxxx|HOME|xxxxx

Simone

[ITWhisperer]

REGEX = (?:SIMONE|MARCO|SIMONE.*HOME) This part is irrelevant since xxxxx|xxxxx|SIMONE|xxxxx|HOME|xxxxx will match with the first alternative

[simo]

sorry my example is incorrect

REGEX = (?:SIMONE|MARCO|DANIELE.*HOME)

xxxxx|xxxxx|DANIELE|xxxxx|HOME|xxxxx
xxxxx|xxxxx|DANIELE|xxxxx|OFFICE|xxxxx
xxxxx|xxxxx|xxxxx|SIMONE|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|VALERIO|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|SILVIA|xxxxx|xxxxx|xxxxx
xxxxx|xxxxx|xxxxx|MARCO|xxxxx|xxxxx|xxxxx

[ITWhisperer]

That is the correct expression to match the bold events

[simo]

ok, but it is possible to filter without regex?

[ITWhisperer]

Yes, you just make the REGEX match anything

REGEX = .

[PickleRick]

Are you sure your data is properly broken into events?

It looks as if you were processing a huge chunk of data at once thus hitting too many matches.