topic Having some trouble with an infinite forwarding loop - Windows Event Logs in Getting Data In

Having some trouble with an infinite forwarding loop - Windows Event Logs

janderson19 — Fri, 22 Jul 2016 17:11:06 GMT

Hello

I'm having a problem with Windows Event logs coming into Splunk.

Windows Events log every time that the Forwarder connects to the Indexer, generating about 25GB of data per day. I had this fixed, but at some point yesterday the indexer stopped sending data containing the indexer's IP address to nullQueue. I don't know why this stopped filtering data, because nobody logged into the server yesterday, and the props.conf and transforms.conf look the same.

Here is the props.conf

 [host::$ComputerName.*]
 TRANSFORMS-null = setnull2,setparsing2

and the transforms:

 [setparsing2]
 REGEX = .
 DEST_KEY = queue
 FORMAT = indexQueue

 [setnull2]
 REGEX = .*(9997).*
 DEST_KEY = queue
 FORMAT = nullQueue

Here is an example of 99.98% of the data coming in:

07/22/2016 12:43:04 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=5156
EventType=0
Type=Information
ComputerName=$ComputerName
TaskCategory=Filtering Platform Connection
OpCode=Info
RecordNumber=18494355
Keywords=Audit Success
Message=The Windows Filtering Platform has permitted a connection.

Application Information:
Process ID: 3860
Application Name: \device\harddiskvolume2\program files\splunkuniversalforwarder\bin\splunkd.exe

Network Information:
Direction: Outbound
Source Address: $srcIP
Source Port: $Port
Destination Address: $dstIP
Destination Port: 9997
Protocol: 6

Filter Information:
Filter Run-Time ID: 77106
Layer Name: Connect
Layer Run-Time ID: 48

This configuration worked just fine for 2 or 3 months, but stopped working yesterday.

Re: Having some trouble with an infinite forwarding loop - Windows Event Logs

dshpritz — Fri, 22 Jul 2016 18:19:42 GMT

If your forwarders are version 6 or above, you can use the Windows event filtering, which would prevent sending the data to the indexers in the first place. More info:

http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/
https://answers.splunk.com/answers/152131/filter-windows-eventcode-using-blacklist-and-whitelist.html#answer-152132

I feel like this would be a more effective way of accomplishing the same goal, if you are trying to filter based on the event code.

Edit: This can also be used to filter based on information within the Windows event.

Re: Having some trouble with an infinite forwarding loop - Windows Event Logs

woodcock — Fri, 22 Jul 2016 18:22:46 GMT

If (and that is a BIG IF because you have not told us how/what you are trying to filter, leaving us no choice but to guess) you are trying to throw away any event that contains the number 9997 anywhere in it, then you can make this change and it will work;

props.conf:

 TRANSFORMS-null = setnull2

Be aware that your RegEx is probably sub-optimal and this would probably be better:

(:?[^0-9]|^)9997(?:[^0-9]|$)

I think probably you actually should be using this instead (but who can really know?):

Destination\s+Port:\s*9997(?:[^0-9]|$)

Re: Having some trouble with an infinite forwarding loop - Windows Event Logs

janderson19 — Fri, 22 Jul 2016 18:24:01 GMT

I have looked into this previously, but I am trying to filter by either destination port or destination Address. In the configuration in the OP, I'm filtering by source port.

Re: Having some trouble with an infinite forwarding loop - Windows Event Logs

dshpritz — Fri, 22 Jul 2016 18:27:00 GMT

Have you tried something like:

[WinEventLog://Security]
blacklist1 = Destination\s+Port:\s+9997

Re: Having some trouble with an infinite forwarding loop - Windows Event Logs

janderson19 — Fri, 22 Jul 2016 18:43:43 GMT

This worked great, thanks!