topic Re: Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer in Getting Data In

Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer

gn694 — Tue, 29 Sep 2020 14:57:22 GMT

I have some TSV files that I am forwarding with a Universal Forwarder.
I have props.conf configured on the UF with the following for the sourcetype:
FIELD_DELIMITER = \t
HEADER_FIELD_LINE_NUMBER = 1

That has worked great. But now I have a need to drop some events so they do not get indexed.
On the Indexer I have configured the following for the sourcetype in props.conf:
[]
TRANSFORMS-null = drop_batchrequests
...and in transforms.conf:
[drop_batchrequests]
REGEX = batchRequest
DEST_KEY = queue
FORMAT = nullQueue

At first it was not working, I was still getting events that contain batchRequest. So I temporarily removed the structured data configuration on the Universal Forwarder (shown above) and the transform worked as desired - batchRequest events were no longer indexed.... But now the tsv format and field recognition was not there...

So I tried to configure everything in one place. On the Indexer I specified the structured data config in props.conf using FIELD_DELIMITER and FIELD_NAMES (since I can't use HEADER_FIELD_LINE_NUMBER on the Inedxer.) The result of that was the batchRequests events were not indexed, but the fields (from the header row) still were not extracted.

Am I doing something wrong? Or is there some reason why these configurations (TSV/structured data field recognition and dropping certain events to the nullQueue) on the same sourcetype will not work together? I can get each to work independently - but not together.

Re: Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer

mattymo — Wed, 19 Jul 2017 23:28:04 GMT

Hi gn694,

Try configuring the INDEXED_EXTRACTIONS props and the filtering on the UF. Structured data is the only data that a UF can complete filtering on.

This should be helpful in this scenario:

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Caveats_for_routing_and_filtering_structured_data

Re: Structured data (TSV) configured on UNiversal Forwarder with Transform applied on Indexer

gn694 — Thu, 20 Jul 2017 17:57:20 GMT

I added the transforms to the Universal Forwarder to send the unwanted stuff to the nullQueue and it is now working as I need it to. I didn't think that would work (even on structured data) but it seems that it does.

thank you!