How does splunk process transforms.conf ?

[jeffking]

Hello, I have a couple of questions -

I have a splunk forwarder with the SCOM app forwarding events to my splunk indexer. I have a lot of events forwarding and noticed that it takes a few hours before the events show up via search. How can I determine if this is due to the indexer still processing events or the forwarder being slow sending the events to the indexer?

I'm trying to cut down on the events by filtering out some unneeded events. I don't need any events from this sourcetype that contain "Failure Code: 0x19". Here's what I put in my props.conf & transforms.com

props.conf:

[SCOM:Events:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = (?m)^.*Failure\s+Code:\s+0x19.*$
DEST_KEY = queue
FORMAT = nullQueue

How can I tell this is working if the indexer is still processing events? Does it process the event when it indexes or when it receives the event from the forwarder? Does the regex look okay to you?

Thanks in advance!

[lukejadamec · Accepted solution]

You can test the regex with a search in the search app.

sourcetype=SCOM:Events:Security | regex _raw=(?m)^.*Failure\s+Code:\s+0x19.*$

The Transforms are processed on the indexer unless the forwarder is a heavy forwarder that is cooking the data in which case it is done on the heavy forwarder.

[lukejadamec]

You are welcome. Feel free to accept the answer:)

[jeffking]

Thanks for the help, lukejadamec. I set the transform on the heavy forwarder and that seems to do the trick. Also, by stopping the heavy forwarder I noticed events stopped coming into the indexer. So that answers my first question about the bottleneck. It's slow from the forwarder, and adding that nullqueue I believe will help.

[lukejadamec]

You should put the transform on the heavy forwarder that way if you should make a change in the future to the forwarder function you won't loose this piece.
With transforms, they are done only once - once cooked it cannot be recooked. In other words, you can't transform in both places.

[jeffking]

Thanks. That is how I built the regex query and it works in the search app. The forwarder is a heavy forwarder, but I don't think the SCOM app is cooking the data, just forwarding it. I think I need to do some testing where the transforms.conf is being processed.