Splunk Search

How to filter logs from the source with a universal forwarder?

Log_wrangler
Builder

Hi,

I have UFs on a few ec2 aws instances, reading logs from /temp.

I want to regex and only send logs containing ERROR and WARN on to the HF and then on to the indexers.

I want to the filter to occur closest to the source to reduce the amount of data being sent.

Is it possible to regex in the inputs.conf of the UF? If so please explain.

Thank you

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.
It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).
You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

FrankVl
Ultra Champion

What is generating those logs? Any chance you can have that write error/warn events to a separate file?

0 Karma

xpac
SplunkTrust
SplunkTrust

Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.
It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).
You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Log_wrangler
Builder

Thanks for confirming, I was pretty sure I would need to use the HF to regex and send, just wanted to make sure.

0 Karma

xpac
SplunkTrust
SplunkTrust

Be aware that data pre-processed by an HF is much larger on the network than what an UF sends. You might add complexity without reducing the data stream, plus you have to do all the parsing work on the HF then 🙂

0 Karma

Log_wrangler
Builder

yes that is a good point.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...