Why is Universal Forwarder unable to process props.conf configuration for structured data?

[bsanch2]

I have a customer that wants to index psv files with headers. If I omit the props.conf file on the Universal Forwarder (UF), the entire psv file gets indexed as one event without any parsing. I have a props.conf on the indexer, but it's my understanding that the indexer does not parse forwarded structured data. However, when I add the props.conf to the UF's, no data is indexed. I have tried with UF versions 6.1.2, and 6.4 running on Linux and Sun. My inputs.conf and props.conf on the UF's are as follows:

inputs.conf

[monitor:///tmp/testmetrics*.txt]
crcSalt = 
sourcetype = test_pri
index = test
disabled = 0

props.conf

[test_pri]
FIELD_DELIMITER=|
HEADER_FIELD_DELIMITER=|
HEADER_FIELD_LINE_NUMBER=1
INDEXED_EXTRACTIONS=psv
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=DATETIME
TIME_FORMAT=%Y%m%d/%H%M%S
KV_MODE=none

The data is in this format with CRLF terminations after each line:

col1|col2|col3
row1|row11|row1111
row2|row22|row222
row3|row33|row333

splunkd.log:

12-02-2016 15:02:13.567 -0500 INFO  WatchedFile - Will begin reading at offset=0 for file='/tmp/testmetrics.txt'.
12-02-2016 15:03:02.914 DEBUG TailingProcessor - File state notification for path='/tmp/testmetrics.txt' (first time).
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/etc/splunk.version' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/log/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/log/splunk/splunkd.log' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/spool/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/spool/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Item '/tmp/testmetrics.txt' matches stanza: /tmp/testmetrics*.txt.
12-02-2016 15:03:03.059 DEBUG TailingProcessor -   Will use CRC salt='/tmp/testmetrics.txt' for this source.
12-02-2016 15:03:03.059 DEBUG FilesystemFilter - Testing path=/tmp/testmetrics.txt(real=/tmp/testmetrics.txt) with global blacklisted paths
12-02-2016 15:03:03.059 DEBUG TailReader -   Will attempt to read file: /tmp/testmetrics.txt.
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG FileClassifierManager - Finding type for file: /tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|test_pri
12-02-2016 15:03:03.059 DEBUG WatchedFile - Storing pending metadata for file=/tmp/testmetrics.txt, sourcetype=test_pri, charset=UTF-8
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|45
12-02-2016 15:03:03.060 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/tmp/testmetrics.txt|host::testhost|test_pri|45 ...
12-02-2016 15:03:03.060 DEBUG VerboseCrc -   Checksumming salt_data="/tmp/testmetrics.txt".
12-02-2016 15:03:03.060 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|46
12-02-2016 15:03:03.060 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/tmp/testmetrics.txt|host::testhost|test_pri|46 ...
12-02-2016 15:03:03.060 DEBUG TailReader - About to read data (Opening file: /tmp/testmetrics.txt).
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=0
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=0
12-02-2016 15:03:03.060 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|46
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=14598
12-02-2016 15:03:03.060 DEBUG WatchedFile - Reached EOF: fname=/tmp/testmetrics.txt fishstate=key=0x915b2ffd0a19e405 sptr=14598 scrc=0xf4eb0f294d1af3b2 fnamecrc=0x5fae16cea4aef038 modtime=1480708933
12-02-2016 15:03:03.060 DEBUG FilesystemChangeWatcher - inotify doing infrequent backup polling for healthy path="/tmp/testmetrics.txt"

Thanks.

[bsanch2 · Accepted solution]

I was able to fix the issue by adding the following stanza to the default-mode.conf file under the SplunkUniversalForwarder app.

[pipeline:structuredparsing]
disabled = false

The problem was that the customer included this app under his Deployment apps, and he was using an old version of this file (5.0.2), so when the app was deployed it overwrote the 6.x version that did include this entry. Parsing structured data from the UF did not get introduced until version 6, so the old file didn't include that stanza.

[masonmorales]

UFs can't perform parsing. You need to put your props.conf file on the indexer. Check out: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

[bsanch2]

Thanks for the response Mason, but the data I am working with is structured, and the UF should have some parsing capabilities when it comes to parsing headers for structured data. In the link you provided there is a section called Structured Data parsing that links to this document:
http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime#Forwa
Under the "Forward data extracted from structured data files - Caveats" section it mentions the following:
"The forwarded data must arrive at the indexer already parsed. To achieve this, you must also set up props.conf on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Universal forwarders are capable of performing these tasks solely for structured data. See "Forward data extracted from header files" earlier in this topic."

Also, I do have the props.conf on the indexer too, I figured it wouldn't hurt if I left it there.