How to filter logs from the source with a universal forwarder?

[Log_wrangler]

Hi,

I have UFs on a few ec2 aws instances, reading logs from /temp.

I want to regex and only send logs containing ERROR and WARN on to the HF and then on to the indexers.

I want to the filter to occur closest to the source to reduce the amount of data being sent.

Is it possible to regex in the inputs.conf of the UF? If so please explain.

Thank you

[FrankVl]

What is generating those logs? Any chance you can have that write error/warn events to a separate file?

[xpac · Accepted solution]

Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.
It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).
You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

[Log_wrangler]

Thanks for confirming, I was pretty sure I would need to use the HF to regex and send, just wanted to make sure.

[xpac]

Be aware that data pre-processed by an HF is much larger on the network than what an UF sends. You might add complexity without reducing the data stream, plus you have to do all the parsing work on the HF then 🙂

[Log_wrangler]

yes that is a good point.