Configuration Validation: Routing and Forwarding

[MasteringIT]

I am building a new Splunk environment, and due to the number of clients we have, we are building a simple distributed environment that consists of 1 Heavy Forwarder, universal forwarders all pointing to the Heavy Forwarder, and 1 indexer.

I would like the heavy forwarder to only forward certain events on to the indexer. Based upon my research (Route and Filter Data ) I have built the below configurations.

outputs.conf

[tcpout]
indexAndForward = 1

[tcpout:indexerGroup]
server = <ip address>:9997

props.conf

[default]
TRANSFORMS-allNull = setnull

[WinEventLog:Security]
TRANSFORMS-WinEventLog=setNull,SendToUserAccountIndex,SendToGroupAccountIndex

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[SendToUserAccountIndex]
REGEX = ^EventCode=(4720|4722|4725|4738)
DEST_KEY = _TCP_ROUTING
FORMAT = indexerGroup

[SendToGroupAccountIndex]
REGEX = ^EventCode=(4727|4730|4728)
DEST_KEY = _TCP_ROUTING
FORMAT = indexerGroup

Unfortunately, even though the specified events are showing in my Windows Event Logs, and the universal forwarder is forwarding the events properly, the events are not showing in the locally indexed items of forwarded items. It is also not showing on the destination server. If I remove the default stanza from props.conf every event populates, including those events that do not meet the criteria listed above. Based upon this, I believe there is an issue with my source types. 

My environment is Splunk Enterprise Version 8.0.3 and and installed on Windows Server 2016. 

[PickleRick]

If I remember correctly, there was some issue with key=value regexes. I think I needed to escape the equals sign with a backslash for the regex to work. You might try that.

Anyway, why don't you use white/blacklisting on thr source UF-s? This way you filter early and don't put unnecessary load on HF.

[MasteringIT]

I will give escaping the equals a shot. 

Regarding the Universal Forwarders, its due to a mandate by our cyber team, that we must maintain a record of each and every log, and can not dispose of any. This is for the "Critical" logs that they want to review daily, whereas all other logs are going to get forwarded to our syslog server by this heavy forwarder. I didn't want to continue setting up additional features, until I could prove this worked.

[PickleRick]

I didn't notice that you're separating two streams of data 🙂

[MasteringIT]

As of right now, I'm not. Once I prove that Splunk is capable of separating the "Critical" logs, I will add an additional tcpout stanza pointing to my syslog server, as well as replacing the current setnull transform to go to the syslog server. In this case, I'm trying to limit the complexities of troubleshooting, by validating and verifying the configuration in stages. There isn't much point in building the full configuration until this portion is working. 

Also, updated my REGEX expression to the following, with no changes to the current behavior:
REGEX = ^EventCode\=(EventCode1|EventCode2|EventCode3)

[PickleRick]

I can show you how I have it done. In my case it's filtering events from a single source because they're being pulled with WMI and with WMI you can't white/blacklist on UF.

props.conf:

[WMI:WinEventLog:Application] 
TRANSFORMS-routing = AAA_default_drop,AAA_index

transforms.conf:

[AAA_default_drop]
REGEX=(?m)^ComputerName=server\d{2}.domain\.com
DEST_KEY=queue
FORMAT=nullQueue

[AAA_index]
REGEX=(?s)EventCode\=10[345][\n\r].*ComputerName\=server\d{2}\.domain\.com
DEST_KEY=queue
FORMAT=indexQueue

As you can see, for some of our servers matching the name serverXX.domain.com and patching the EventCode 103, 104 or 105, I'm sending the events to index and I discard all the remaining events from those hosts.

I don't remember however (I only have config dump at hand, I don't have access to the servers to check it at the moment) whether we send the events from windows to splunk rendered as plain text or xml. But maybe that'll point you somewhere.

[MasteringIT]

Based upon your config, I went and troubleshot a little further. It looks like my issue is actually related to the setNull stanza in transforms.conf.
I moved setNull to the end of the transform list in props.conf, and now events are routing properly, but there is no filtering happening (Event codes I specified and event codes I didnt specify are being routed). To troubleshoot why this happened, I went to my indexer, and tried separating EventCodes into separate indexes using the same REGEX as above, while keeping setNull to the end of my list. In doing so, the data was going to the proper index, but if it didnt meet any of the Event Code criteria, it got dumped into the main index, even though I have the setNull stanza in place. This behavior only applied while setNull was at the end of the props.conf transform list. (it is still the first entry in transforms.conf). This changes to not indexing any events at all once setNull is moved to the beginning of the transform list. 

Upon reviewing the splunk documentation (specifically Route and Filter Data), I noticed there is never a mix of DEST_KEY in any props.conf stanza transform lists. 

So, the question now is: Can I only specify one type of DEST_KEY in transforms.conf per stanza in props.conf? If so, what is the suggested way around this limitation?

[PickleRick · Accepted solution]

Ehhh. I again read too quickly and missed some of the information 😉

My example worked only on queues so it firstly assigned nullQueue to all events and then selectively re-assigned some events back to indexQueue (regexes aren't that great in negation, so it's the easiest way).

In this case you can add additional "whitelist" transforms after the initial nullQueue assignment to reroute other event classes back to indexQueue.

I don't know why you touch the _TCP_ROUTING key at all. If the event is sent to nullQueue, you don't route it anymore because it's gonna get dropped. So it makes sense only for the events passing further down the indexing queue. Changing _TCP_ROUTING would make sense only if you wanted to send various types of events to various destinations (but not for deciding whether you process it at all).

In that case you'd have to do it like that

1) send all to nullQueue

2) "whitelist" some events by sending them to indexQueue

3) if you want to redirect some of the events you just whitelisted to another destination, do a transform modifying _TCP_ROUTE

See the diagram 3 on https://wiki.splunk.com/Community:HowIndexingWorks

The place within transforms.conf of particular stanza doesn't matter. It's the order in props.conf which matters.

With a single transform stanza you can only write to a single field. So in order to modify two fields you'd have to have two separate stanzas.

And for debugging it's usually helpful to leave the routing and splunk internal fields alone at first - just define a custom indexed field and check whether your regexes match by writing different values there.