topic Re: How to edit props.conf to exclude headers in CSV files from getting indexed? in Getting Data In

How to edit props.conf to exclude headers in CSV files from getting indexed?

SirHill17 — Tue, 22 Nov 2016 12:17:01 GMT

Hi,

I have a CSV file with header that is monitored by Splunk. Rows are correctly read, but the headers are also included as an event row. I just want to have the header extracted as the field names (which already works at the same time).

I tried several ideas using props.conf without any success. I also had a look to the similar questions already asked by other users.

My last props.conf looks like:

[mysourcetype]
INDEXED_EXTRACTIONS = csv
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER = ","
FIELD_DELIMITER = ","
FIELD_HEADER_REGEX = hostname,SCSI logical unit,DeviceID,SCSIBus,SCSIPort,SCSITargetId

I hope someone can help sort this out.

Thanks,

SirHill

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

gcusello — Tue, 22 Nov 2016 12:27:00 GMT

Hi SirHill17,
to exclude header from indexing you have to insert in your props.conf the following line

PREAMBLE_REGEX = <regex>

This attribute specifies a regular expression which allows Splunk to ignore these preamble lines, based on the pattern specified.
for other information see https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Propsconf

Bye.
Giuseppe

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

SirHill17 — Tue, 22 Nov 2016 12:38:01 GMT

Just tried adding that to my current props.conf but now it index the entire csv as one event (including the header).

props.conf is defined at the indexers level (master-node), not at the forwarder level but I don't think it changes anything.

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

gcusello — Tue, 22 Nov 2016 12:45:41 GMT

Try

FIELD_HEADER_REGEX=your_regex

Bye.
Giuseppe

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

SirHill17 — Tue, 22 Nov 2016 12:54:22 GMT

As per my inital question, I already tried that and everything here:
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Extractfieldsfromfileswithstructureddata

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

gcusello — Tue, 22 Nov 2016 13:01:10 GMT

If none of the previous options correctly runs, you could filter your header in this way:
props.conf

[your_sourcetype]
TRANSFORMS-set-remove_headers=set_OK,set_nullqueue

transforms.conf

[set_nullqueue]
REGEX=your_header_regex
DEST_KEY=queue
FORMAT=nullQueue

[set_OK]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

woodcock — Sun, 12 Mar 2017 06:28:15 GMT

Did you deploy this file to your FORWARDER (not your indexers) and did you restart splunkd there?

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

gcusello — Sun, 12 Mar 2017 09:08:02 GMT

Did you tried to insert props.conf in your Forwarder?
Bye.
Giuseppe

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

woodcock — Mon, 13 Mar 2017 19:50:47 GMT

YOU MUST DEPLOY THIS ON YOUR FORWARDER. That is the problem.

Re: How to edit props.conf to exclude headers in CSV files from getting indexed?

DUThibault — Tue, 29 Sep 2020 17:56:22 GMT

I have had no success with the PREAMBLE_REGEX and HEADER_FIELD_LINE_NUMBER clauses... But this solution works: https://answers.splunk.com/answers/206718/how-to-pull-out-a-header-before-indexing.html ---It's a workaround, sadly, but until PREAMBLE_REGEX and HEADER_FIELD_LINE_NUMBER are fixed, that's all we have.