Getting Data In

Why does the csv sourcetype work for upload but not via the forwarder?

ddrillic
Ultra Champion

The following sourcetype works fine when we upload a file against this sourcetype, but via the forwarder the csv fields are not being detected?

[incidentinfo]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
Tags (3)
0 Karma
1 Solution

acharlieh
Influencer

I'm guessing that this props.conf isn't present on your forwarders as well and that's causing you issues. The thing to note here is the line:

INDEXED_EXTRACTIONS = csv

This means that your linebreaking and extractions of fields actually happens on your universal forwarder (where the header row of your CSV file is located) and not your indexers... see the diagram on (especially the structured parsing pipeline): https://wiki.splunk.com/Community:HowIndexingWorks

Being a custom sourcetype, your forwarders need to know about it otherwise they won't perform the index time extractions you're expecting that they should be doing. There are some other side effects to this as well such as what is mentioned in this Splunk Reaction.

View solution in original post

acharlieh
Influencer

I'm guessing that this props.conf isn't present on your forwarders as well and that's causing you issues. The thing to note here is the line:

INDEXED_EXTRACTIONS = csv

This means that your linebreaking and extractions of fields actually happens on your universal forwarder (where the header row of your CSV file is located) and not your indexers... see the diagram on (especially the structured parsing pipeline): https://wiki.splunk.com/Community:HowIndexingWorks

Being a custom sourcetype, your forwarders need to know about it otherwise they won't perform the index time extractions you're expecting that they should be doing. There are some other side effects to this as well such as what is mentioned in this Splunk Reaction.

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

Wow - illuminating. So, how do we pass the proper information to the forwarder, which is completely in the dark about my csv expectations?

0 Karma

acharlieh
Influencer

Basically you just need to take a copy of this props.conf entry and push it out to the applicable forwarder(s) as well. How you get it there is a function of "How do you deliver other applicable configuration (like inputs.conf for example) to your forwarders, and restart them to pick up the changes?"

(Unfortunately that's a function of your environment as there are a lot of options for that, from an app pushed with Deployment Server, to some Configuration management solution (Chef, Puppet, Salt, Ansible, CFEngine, SCCM, and the like), or even just scp and manual command line jockeying...)

ddrillic
Ultra Champion

Interesting!!

0 Karma

ddrillic
Ultra Champion

But @acharlieh, in the Input phase by the forwarder, the data is handled as a stream and during the parsing phase, by the indexer, the data is broken into events. That's what my beloved admin guide says ; -)

0 Karma

ddrillic
Ultra Champion

Gorgeous @acharlieh!!!

Configuration parameters and the data pipeline

The link explains exactly what @acharlieh said about the parsing stage -

-- indexer
heavy forwarder
light/universal forwarder (in conjunction with the INDEXED_EXTRACTIONS attribute only)

Meaning, INDEXED_EXTRACTIONS is an exception and needs to be on the props.conf of the forwarder.

0 Karma

ddrillic
Ultra Champion

The working configuration ended as -

INDEXED_EXTRACTIONS = CSV
TIMESTAMP_FIELDS = Date_Time
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
category = Structured
disabled = false
pulldown_type = true
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is this sourcetype in the props.conf file on your indexers?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ddrillic
Ultra Champion

It is. It's a single server and this stanza is in props.conf.

0 Karma

ddrillic
Ultra Champion

Interesting, our sales engineer suggested -

[incidentinfo]
INDEXED_EXTRACTIONS = csv
TIMESTAMP_FIELDS = Date_Time
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER = \,
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Let's see..

0 Karma

ddrillic
Ultra Champion

Meanwhile, as we are stuck, I wonder how we can debug such a scenario. As the events flow through the parsing phase, can we trace them in any log, by any chance?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...