Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer

[gn694]

I have some TSV files that I am forwarding with a Universal Forwarder.
I have props.conf configured on the UF with the following for the sourcetype:
FIELD_DELIMITER = \t
HEADER_FIELD_LINE_NUMBER = 1

That has worked great. But now I have a need to drop some events so they do not get indexed.
On the Indexer I have configured the following for the sourcetype in props.conf:
[]
TRANSFORMS-null = drop_batchrequests
...and in transforms.conf:
[drop_batchrequests]
REGEX = batchRequest
DEST_KEY = queue
FORMAT = nullQueue

At first it was not working, I was still getting events that contain batchRequest. So I temporarily removed the structured data configuration on the Universal Forwarder (shown above) and the transform worked as desired - batchRequest events were no longer indexed.... But now the tsv format and field recognition was not there...

So I tried to configure everything in one place. On the Indexer I specified the structured data config in props.conf using FIELD_DELIMITER and FIELD_NAMES (since I can't use HEADER_FIELD_LINE_NUMBER on the Inedxer.) The result of that was the batchRequests events were not indexed, but the fields (from the header row) still were not extracted.

Am I doing something wrong? Or is there some reason why these configurations (TSV/structured data field recognition and dropping certain events to the nullQueue) on the same sourcetype will not work together? I can get each to work independently - but not together.

[mattymo · Accepted solution]

Hi gn694,

Try configuring the INDEXED_EXTRACTIONS props and the filtering on the UF. Structured data is the only data that a UF can complete filtering on.

This should be helpful in this scenario:

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Caveats_for_routin...

[gn694]

I added the transforms to the Universal Forwarder to send the unwanted stuff to the nullQueue and it is now working as I need it to. I didn't think that would work (even on structured data) but it seems that it does.

thank you!