Why does the csv sourcetype work for upload but not via the forwarder?

[ddrillic]

The following sourcetype works fine when we upload a file against this sourcetype, but via the forwarder the csv fields are not being detected?

[incidentinfo]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

[acharlieh · Accepted solution]

I'm guessing that this props.conf isn't present on your forwarders as well and that's causing you issues. The thing to note here is the line:

INDEXED_EXTRACTIONS = csv

This means that your linebreaking and extractions of fields actually happens on your universal forwarder (where the header row of your CSV file is located) and not your indexers... see the diagram on (especially the structured parsing pipeline): https://wiki.splunk.com/Community:HowIndexingWorks

Being a custom sourcetype, your forwarders need to know about it otherwise they won't perform the index time extractions you're expecting that they should be doing. There are some other side effects to this as well such as what is mentioned in this Splunk Reaction.

[ddrillic]

@acharlieh, just posted a follow-up question at Are there inconsistencies in behavior with the need for INDEXED_EXTRACTIONS?

[ddrillic]

Wow - illuminating. So, how do we pass the proper information to the forwarder, which is completely in the dark about my csv expectations?

[acharlieh]

Basically you just need to take a copy of this props.conf entry and push it out to the applicable forwarder(s) as well. How you get it there is a function of "How do you deliver other applicable configuration (like inputs.conf for example) to your forwarders, and restart them to pick up the changes?"

(Unfortunately that's a function of your environment as there are a lot of options for that, from an app pushed with Deployment Server, to some Configuration management solution (Chef, Puppet, Salt, Ansible, CFEngine, SCCM, and the like), or even just scp and manual command line jockeying...)

[ddrillic]

Interesting!!

[ddrillic]

But @acharlieh, in the Input phase by the forwarder, the data is handled as a stream and during the parsing phase, by the indexer, the data is broken into events. That's what my beloved admin guide says ; -)

[ddrillic]

Gorgeous @acharlieh!!!

Configuration parameters and the data pipeline

The link explains exactly what @acharlieh said about the parsing stage -

-- indexer
heavy forwarder
light/universal forwarder (in conjunction with the INDEXED_EXTRACTIONS attribute only)

Meaning, INDEXED_EXTRACTIONS is an exception and needs to be on the props.conf of the forwarder.

[ddrillic]

The working configuration ended as -

INDEXED_EXTRACTIONS = CSV
TIMESTAMP_FIELDS = Date_Time
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
category = Structured
disabled = false
pulldown_type = true

[richgalloway]

Is this sourcetype in the props.conf file on your indexers?

[ddrillic]

It is. It's a single server and this stanza is in props.conf.

[ddrillic]

Interesting, our sales engineer suggested -

[incidentinfo]
INDEXED_EXTRACTIONS = csv
TIMESTAMP_FIELDS = Date_Time
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER = \,
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Let's see..

[ddrillic]

Meanwhile, as we are stuck, I wonder how we can debug such a scenario. As the events flow through the parsing phase, can we trace them in any log, by any chance?