Do we need props.conf on the indexer when indexing a csv file?

[ddrillic]

We use the following props.conf for csv files -

[<sourcetype>]
disabled = false
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = CSV
FIELD_NAMES = <comma separated field names>
TIMESTAMP_FIELDS = <time stamp field>
TIME_FORMAT = <format of time field>

We place it on the forwarder - do we need to place it on the indexer as well?

props.conf.spec says -

-- Structured Data Header Extraction and configuration

This setting applies at input time, when data is first read by Splunk software, such as on a forwarder that has configured inputs acquiring the data.

[PowerPacked]

@ddrillic

Did you put this props on universal forwarder ?

Does universal forwarder can handle "SHOULD_LINEMERGE", TIME_FORMAT attributes ?

Thanks

[ddrillic]

Interesting thing. When searching the props.conf page for input time

We can see the ones that are strictly the input time ones, such as HEADER_MODE, the beloved CHARSET, obviously INDEXED_EXTRACTIONS, CHECK_FOR_HEADER and the famous NO_BINARY_CHECK.

When searching for SHOULD_LINEMERGE the page doesn't say explicitly in which phase it takes place.

[markusspitzli]

no it doesn't.
The universalf forwarder handles the input phase.
SHOULD_LINEMERGE and TIME_FORMAT are part of the parsing phase which does Heavyforwarders or Indexers

[PowerPacked]

Thanks for reply

Please take a look at this lines in documenation which is confusing.

If you want to forward fields that you extract from structured data files to another Splunk instance, you must configure the props.conf settings that define the field extractions on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Performing these actions on the instance that indexes the data will have no effect, as the forwarded data must arrive at the indexer already parsed.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/Extractfieldsfromfileswithstructureddata#Fie...

Thanks

[markusspitzli]

On the forwarder you need the following configuration:

[<sourcetype>]
disabled = false
INDEXED_EXTRACTIONS = CSV
FIELD_NAMES = <comma separated field names>

Indexers or Heavyforwarders, which are responsible for the parsing phase need this configuration:

[<sourcetype>]
disabled = false
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = <time stamp field>
TIME_FORMAT = <format of time field>

[vr2312]

@ddrillic Ah I get you now. You can still place it on the IDX as part of your keeping things Organized process. But the process work would still happen at the FW thus giving but no impact on your IDX.

As much as I have dealt with support (the top tier) and PS, they would want the processing to be occurring at the FW due to a high resource environment like ours. But when it comes to an aesthetic purpose, keeping it on IDX won't have much impact. We have been doing the same.

[ddrillic]

Right, for the csv case, it simply won't work if the configuration is not on the forwarder...

[vr2312]

Yup. @ddrillic

[vr2312]

Hello @ddrillic , if you have an Intermediate HF between the UF and IDX, then you would not need to place in the IDX. However, if the IDX does the data parsing and data indexing, you would need these.

[ddrillic]

Thank you @vr2312 - my question is whether we need/should place the props.conf on the indexer in addition to the forwarder.

For one thing, if we place it only on the forwarder, then it's harder to see all the indexer configurations together, as most would end up on the indexers and several are on the forwarders. So, from best practices perspective, it's interesting.