https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375562#M110357 <P>Hi, </P> <P>I have UFs on a few ec2 aws instances, reading logs from /temp.</P> <P>I want to regex and only send logs containing ERROR and WARN on to the HF and then on to the indexers.</P> <P>I want to the filter to occur closest to the source to reduce the amount of data being sent.</P> <P>Is it possible to regex in the inputs.conf of the UF? If so please explain.</P> <P>Thank you</P> Fri, 04 May 2018 15:45:31 GMT Log_wrangler 2018-05-04T15:45:31Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375562#M110357 <P>Hi, </P> <P>I have UFs on a few ec2 aws instances, reading logs from /temp.</P> <P>I want to regex and only send logs containing ERROR and WARN on to the HF and then on to the indexers.</P> <P>I want to the filter to occur closest to the source to reduce the amount of data being sent.</P> <P>Is it possible to regex in the inputs.conf of the UF? If so please explain.</P> <P>Thank you</P> Fri, 04 May 2018 15:45:31 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375562#M110357 Log_wrangler 2018-05-04T15:45:31Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375563#M110358 <P>Hey, the UF has no real concept of lines, or events, it just see's blocks of data and sends that to the indexers for processing.<BR /> It's therefore not possible to regex-filter on the inputs (with a few rare exemptions, for example on Windows event logs).<BR /> You'll either have to built a scripted input that reads the files and filters them for you, or install an HF, or just send the logs and have the first HF/indexer they reach do the filtering.</P> <P>Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P> Fri, 04 May 2018 17:22:12 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375563#M110358 xpac 2018-05-04T17:22:12Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375564#M110359 <P>Thanks for confirming, I was pretty sure I would need to use the HF to regex and send, just wanted to make sure.</P> Fri, 04 May 2018 17:26:55 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375564#M110359 Log_wrangler 2018-05-04T17:26:55Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375565#M110360 <P>Be aware that data pre-processed by an HF is much larger on the network than what an UF sends. You might add complexity without reducing the data stream, plus you have to do all the parsing work on the HF then <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P> Fri, 04 May 2018 17:35:36 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375565#M110360 xpac 2018-05-04T17:35:36Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375566#M110361 <P>yes that is a good point.</P> Fri, 04 May 2018 19:54:00 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375566#M110361 Log_wrangler 2018-05-04T19:54:00Z https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375567#M110362 <P>What is generating those logs? Any chance you can have that write error/warn events to a separate file?</P> Mon, 07 May 2018 06:31:01 GMT https://community.splunk.com/t5/Splunk-Search/How-to-filter-logs-from-the-source-with-a-universal-forwarder/m-p/375567#M110362 FrankVl 2018-05-07T06:31:01Z