Field extractions don't work for forwarded input from Universal Forwarder

[jordansamuels_h]

I have a custom source type and field extractions which work perfectly well when indexed locally on the Splunk Enterprise server (indexer + search head). However, when the same type of input is forwarded from a UF, the extractions don't work.

On the Splunk Enterprise server (Splunk 6.1.3 (build 220630) on RH 6.5), /apps/splunk/splunk/etc/system/local/props.conf has this stanza:

[test_pipe_2]
FIELD_DELIMITER = |
FIELD_NAMES = time,c1,c2  
HEADER_FIELD_DELIMITER = |
INDEXED_EXTRACTIONS = psv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1

On the UF (Splunk Universal Forwarder 6.1.3 (build 220630) on RH 6.4), /opt/splunkforwarder/etc/system/local/inputs.conf has this stanza:

[monitor:///tmp/testme]
disabled = false
index = myindex
sourcetype = test_pipe_2

When I put files in /tmp/testme that look like below, I get events in searching for index=myindex and they do have sourcetype test_pipe_2, but the fields c1 and c2 are not extracted.

However, when I put similar text in a file on the Enterprise server (indexer+search head) and index it with sourcetype test_pipe_2, it has the fields.

Sample input:

Wed Sep 10 10:14:01 CDT 2014|apple|pear

Note that I have also tried placing the same stanza in /opt/splunkforwarder/etc/apps/search/local/inputs.conf on the UF, same result.

What am I doing wrong?

Another experiment: when I purposely misspell the sourcetype on the UF inputs.conf, to a non-existent sourcetype, it still gets indexed and just shows up in searches as that sourcetype. I'm not sure what that implies but it seems interesting.

[lguinn2 · Accepted solution]

This syntax is only used to create index-time field extractions in props.conf on a forwarder

FIELD_DELIMITER = |
FIELD_NAMES = time,c1,c2  
HEADER_FIELD_DELIMITER = |
INDEXED_EXTRACTIONS = psv

You can't do this in props.conf on an indexer! (Well, actually you can do this on an indexer but only for data that is collected locally on the indexer - not on data that was forwarded to the indexer.)

On the indexer, you want to create search-time field extractions, using this syntax:

props.conf

[test_pipe_2]
REPORT-r1 = extract-test-pipe-fields
# other stuff is fine

transforms.conf

[extract-test-pipe-fields]
DELIMS = "|"
FIELDS=time,c1,c2

You can put props.conf and transforms.conf in etc/system/local or etc/apps/search/local - it doesn't matter. I would put them both in the same directory to avoid confusion later.

To answer "another experiment" - When you set a sourcetype in inputs.conf, that IS the sourcetype that is assigned. Splunk does not have a concept of an "invalid" sourcetype. The sourcetype is whatever you say it is.

(But you can override the sourcetype that was set on the UF. Although the appropriate fix is to spell it right in inputs.conf!)

[jordansamuels_h]

@lguinn, I'm grateful for your continued attention to my questions. If you're saying that props.conf goes on the UF, I'm not sure I understand this answer , which seems to indicate that it does not. Also, I tried this in a similar situation and it didn't seem to have any effect.

[lguinn2]

props.conf goes on the UF, on the indexer, on the search head - pretty much everywhere! HOWEVER, not all of the settings apply to the UF (or the indexer, etc.). So you may have props.conf on the UF - but certain settings will be ignored on the UF, because it only performs the "input phase." This article will probably clear things up for you: Where do I configure my Splunk settings?

[lguinn2]

Oh, what a typo! You are right, it doesn't go in inputs.conf, it goes in props.conf!!

[I've corrected my comment above - hopefully anyone who reads the whole thread won't be confused]

[jordansamuels_h]

@lguinn - also, seem like http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Extractfieldsfromfileheadersatindextime implies that those directives can go in props.conf, at least in certain contexts.

[jordansamuels_h]

@lguinn now I'm confused again. Your comment "just move original stuff back to inputs.conf on the UF" seems to imply that's were it was in the first place; but I had it in props.conf on the indexer/searcher. The reason it was there, by the way, is that Splunk itself put it there, when I created a new custom sourcetype via Splunk Web. That also makes me confused on second reading of your answer where you say "you can't do this in props.conf!" Splunk itself thinks I can (since it put it there!).

[jordansamuels_h]

Thanks for your quick and accurate response. This solves the problem! And your additional answer makes sense for the second question I snuck in there.

Clearly, I need to read more of the manual, and spend less time tweaking files created from clicking around on the web pages!

[lguinn2]

Oh, or you could just move your original stuff back to props.conf on the UF

FIELD_DELIMITER = |
FIELD_NAMES = time,c1,c2
HEADER_FIELD_DELIMITER = |
INDEXED_EXTRACTIONS = psv

Leaving everything else as it is in props.conf on the indexer.

[edited to correct where I said inputs.conf instead of props.conf]