How do I find and remove strings in logs from the Forwarder?

brandonmcgrath1 — Fri, 14 Sep 2018 07:04:09 GMT

Hello,

I'm trying to send some antivirus logs from the forwarder into Splunk.

The logs I'm sending have a tendency to spam, for example:

13/09/2018 16:06:53 No usable rule found    Blocked 192.168.0.40:53354  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:54 No usable rule found    Blocked 192.168.0.40:52091  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:54 No usable rule found    Blocked 192.168.0.40:49467  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:54 No usable rule found    Blocked 192.168.0.40:53354  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:55 No usable rule found    Blocked 192.168.0.40:52091  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:56 No usable rule found    Blocked 192.168.0.40:53354  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:57 No usable rule found    Blocked 192.168.0.40:52091  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:58 No usable rule found    Blocked 192.168.0.40:56694  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM
13/09/2018 16:06:59 No usable rule found    Blocked 192.168.0.40:56694  192.168.0.30:53 UDP     C:\Windows\System32\dns.exe NT AUTHORITY\SYSTEM

I want to be able to filter out lines in the log that say "No usable rule found".

I've tried adding the following to props.conf which I've copied into [C:\Program Files\SplunkUniversalForwarder\etc\system\local] directory, here is the line I added to props.conf:

[source:\path\to\log\log.txt]
SEDCMD-strip-detail-msg=^.*(listening on the port|[Nn]o usable rule found)*$

I have also tried messing with transform.conf too, but to no avail.

Any ideas guys?

Re: How do I find and remove strings in logs from the Forwarder?

richgalloway — Fri, 14 Sep 2018 11:25:17 GMT

Universal forwarders cannot filter events. You must use a heavy forwarder or put your props.conf settings on your indexers.

The SEDCMD setting is not correct. It must be in the format s/<search regex>/<replacement string>/. Replacing the entire event string will not remove the event. To remove unwanted events, send them to nullQueue like so:

props.conf

[source:\path\to\log\log.txt] 
TRANSFORMS-set = setnull,setparsing

transforms.conf

[setnull]
REGEX = (listening on the port|[Nn]o usable rule found)
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

topic How do I find and remove strings in logs from the Forwarder? in Getting Data In

How do I find and remove strings in logs from the Forwarder?

Re: How do I find and remove strings in logs from the Forwarder?