I have a custom source type and field extractions which work perfectly well when indexed locally on the Splunk Enterprise server (indexer + search head). However, when the same type of input is forwarded from a UF, the extractions don't work.
On the Splunk Enterprise server (Splunk 6.1.3 (build 220630) on RH 6.5), /apps/splunk/splunk/etc/system/local/props.conf has this stanza:
[test_pipe_2] FIELD_DELIMITER = | FIELD_NAMES = time,c1,c2 HEADER_FIELD_DELIMITER = | INDEXED_EXTRACTIONS = psv KV_MODE = none NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = false pulldown_type = 1
On the UF (Splunk Universal Forwarder 6.1.3 (build 220630) on RH 6.4), /opt/splunkforwarder/etc/system/local/inputs.conf has this stanza:
[monitor:///tmp/testme] disabled = false index = myindex sourcetype = test_pipe_2
When I put files in /tmp/testme that look like below, I get events in searching for index=myindex and they do have sourcetype test_pipe_2, but the fields c1 and c2 are not extracted.
However, when I put similar text in a file on the Enterprise server (indexer+search head) and index it with sourcetype testpipe2, it has the fields.
Wed Sep 10 10:14:01 CDT 2014|apple|pear
Note that I have also tried placing the same stanza in /opt/splunkforwarder/etc/apps/search/local/inputs.conf on the UF, same result.
What am I doing wrong?
Another experiment: when I purposely misspell the sourcetype on the UF inputs.conf, to a non-existent sourcetype, it still gets indexed and just shows up in searches as that sourcetype. I'm not sure what that implies but it seems interesting.
This syntax is only used to create index-time field extractions in
props.conf on a forwarder
FIELD_DELIMITER = | FIELD_NAMES = time,c1,c2 HEADER_FIELD_DELIMITER = | INDEXED_EXTRACTIONS = psv
You can't do this in
props.conf on an indexer! (Well, actually you can do this on an indexer but only for data that is collected locally on the indexer - not on data that was forwarded to the indexer.)
On the indexer, you want to create search-time field extractions, using this syntax:
[test_pipe_2] REPORT-r1 = extract-test-pipe-fields # other stuff is fine
[extract-test-pipe-fields] DELIMS = "|" FIELDS=time,c1,c2
You can put
etc/apps/search/local - it doesn't matter. I would put them both in the same directory to avoid confusion later.
To answer "another experiment" - When you set a sourcetype in
inputs.conf, that IS the sourcetype that is assigned. Splunk does not have a concept of an "invalid" sourcetype. The sourcetype is whatever you say it is.
(But you can override the sourcetype that was set on the UF. Although the appropriate fix is to spell it right in
Oh, or you could just move your original stuff back to
props.conf on the UF
FIELD_DELIMITER = |
FIELD_NAMES = time,c1,c2
HEADER_FIELD_DELIMITER = |
INDEXED_EXTRACTIONS = psv
Leaving everything else as it is in
props.conf on the indexer.
[edited to correct where I said
inputs.conf instead of
Thanks for your quick and accurate response. This solves the problem! And your additional answer makes sense for the second question I snuck in there.
Clearly, I need to read more of the manual, and spend less time tweaking files created from clicking around on the web pages!
@lguinn now I'm confused again. Your comment "just move original stuff back to
inputs.conf on the UF" seems to imply that's were it was in the first place; but I had it in props.conf on the indexer/searcher. The reason it was there, by the way, is that Splunk itself put it there, when I created a new custom sourcetype via Splunk Web. That also makes me confused on second reading of your answer where you say "you can't do this in props.conf!" Splunk itself thinks I can (since it put it there!).
@lguinn - also, seem like http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Extractfieldsfromfileheadersatindextime implies that those directives can go in
props.conf, at least in certain contexts.
Oh, what a typo! You are right, it doesn't go in
inputs.conf, it goes in
[I've corrected my comment above - hopefully anyone who reads the whole thread won't be confused]
@lguinn, I'm grateful for your continued attention to my questions. If you're saying that
props.conf goes on the UF, I'm not sure I understand this answer , which seems to indicate that it does not. Also, I tried this in a similar situation and it didn't seem to have any effect.
props.conf goes on the UF, on the indexer, on the search head - pretty much everywhere! HOWEVER, not all of the settings apply to the UF (or the indexer, etc.). So you may have props.conf on the UF - but certain settings will be ignored on the UF, because it only performs the "input phase." This article will probably clear things up for you: Where do I configure my Splunk settings?