How to keep specific events and discard the rest in props.conf and transforms.conf?

[ankithreddy777]

In splunk doc it is mentioned that**

[[[Note**: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue]]]

why we need to put nullqueue transform first and index queue later? Putting setnull transform first do not delete events before going to setparsing transform?

[woodcock]

There are 2 ways to trash events at the indexer:

EITHER:
You can trash the events that match a "drop me" pattern. In this case, simply send the matching events to the nullQueue.
OR:
You can send everything to nullQueue first and then pull back the events that match a "keep me" pattern to the indexQueue.

The last queue location wins.

[bhargavi]

Hi @woodcock @somesoni2 

Could you please help me out here. I have a little different scenario here, but facing similar issue.

We are integrating the json logs via HEC into Splunk Heavy Forwarder. I have tried the below configurations.I am applying the props for the source.

In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.

The configs are like below:

PROPS.CONF --

[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs

TRANSFORMS.CONF --

[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue

[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000

[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000

[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true

[somesoni2]

The transform on the right side is executed first, so if nullQueue is on right side (executed first), it will dump all data to nullQueue as there is no specific regex.