topic Re: Universal Forwarder props.conf and transforms.conf settings in Getting Data In

Universal Forwarder props.conf and transforms.conf settings

kniloo — Wed, 30 Sep 2020 03:23:05 GMT

I am trying to get the output from a python script to indexer. So i added transforms.conf and props.conf under C:\Program Files\SplunkUniversalForwarder\etc\system\local

transforms.conf
[myexternaltable]
REGEX = (.)
external_cmd = addnum.py $1
DEST_KEY = queue
FORMAT = indexQueue

props.conf
[sitescope_daily2_log]
TRANSFORMS-runscript=myexternaltable

But its not working, can anyone please help me with correct settings needs to be done on UF.

Thanks,
Niloo

Re: Universal Forwarder props.conf and transforms.conf settings

richgalloway — Tue, 17 Dec 2019 13:09:02 GMT

The props.conf and transforms.conf files should be installed on the indexer(s), not the UF.

Re: Universal Forwarder props.conf and transforms.conf settings

kniloo — Tue, 17 Dec 2019 13:27:28 GMT

Thanks for the response.
But if we required to parse some data at UF (before sending to indexer) can't we use transforms.conf and props.conf on UF ?

if yes ,can you share the steps as well.

Re: Universal Forwarder props.conf and transforms.conf settings

richgalloway — Tue, 17 Dec 2019 14:28:55 GMT

What is the source of this requirement? Just because it is required does not make it possible (or correct).
The filtering you are trying to do is performed by indexers or heavy forwarders, not universal forwarders. Consider replacing the UF with a HF.

Re: Universal Forwarder props.conf and transforms.conf settings

kniloo — Wed, 30 Sep 2020 03:23:42 GMT

I have moved props.conf and transforms.conf to indexer ,but still its not working.
transforms.conf
[myexternaltable]
REGEX = (.)
external_cmd = testscript.py $1
fields_list = log
DEST_KEY = queue
FORMAT = indexQueue
WRITE_META = true

props.conf
[sitescope_daily2_log]
TRANSFORMS-runscript=myexternaltable

Re: Universal Forwarder props.conf and transforms.conf settings

mikev — Thu, 20 Aug 2020 15:20:43 GMT

I know this is an older post but I believe that you should be using DEST_KEY per the documentation:

DEST_KEY = <KEY>
* NOTE: This setting is only valid for index-time field extractions.
* Specifies where Splunk software stores the expanded FORMAT results in
  accordance with the REGEX match.
* Required for index-time field extractions where WRITE_META = false or is
  not set.
* For index-time extractions, DEST_KEY can be set to a number of values
  mentioned in the KEYS section at the bottom of this file.
  * If DEST_KEY = _meta (not recommended) you should also add $0 to the
    start of your FORMAT setting.  $0 represents the DEST_KEY value before
    Splunk software performs the REGEX (in other words, _meta).
    * The $0 value is in no way derived *from* the REGEX match. (It
      does not represent a captured group.)
* KEY names are case-sensitive, and should be used exactly as they appear in
  the KEYs list at the bottom of this file. (For example, you would say
  DEST_KEY = MetaData:Host, *not* DEST_KEY = metadata:host .)

Re: Universal Forwarder props.conf and transforms.conf settings

didatams — Thu, 20 Aug 2020 15:55:30 GMT

Just an idea.. but if you want to input data from a script.

You can put the script in the bin directory of an app, refer it in the inputs.conf.