Getting Data In

How to edit props.conf to exclude headers in CSV files from getting indexed?

Communicator

Hi,

I have a CSV file with header that is monitored by Splunk. Rows are correctly read, but the headers are also included as an event row. I just want to have the header extracted as the field names (which already works at the same time).

I tried several ideas using props.conf without any success. I also had a look to the similar questions already asked by other users.

My last props.conf looks like:

[mysourcetype]
INDEXED_EXTRACTIONS = csv
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER = ","
FIELD_DELIMITER = ","
FIELD_HEADER_REGEX = hostname,SCSI logical unit,DeviceID,SCSIBus,SCSIPort,SCSITargetId

I hope someone can help sort this out.

Thanks,

SirHill

0 Karma
1 Solution

Esteemed Legend

YOU MUST DEPLOY THIS ON YOUR FORWARDER. That is the problem.

View solution in original post

0 Karma

Contributor

I have had no success with the PREAMBLE_REGEX and HEADER_FIELD_LINE_NUMBER clauses... But this solution works: https://answers.splunk.com/answers/206718/how-to-pull-out-a-header-before-indexing.html ---It's a workaround, sadly, but until PREAMBLE_REGEX and HEADER_FIELD_LINE_NUMBER are fixed, that's all we have.

0 Karma

Esteemed Legend

YOU MUST DEPLOY THIS ON YOUR FORWARDER. That is the problem.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Did you tried to insert props.conf in your Forwarder?
Bye.
Giuseppe

0 Karma

Esteemed Legend

Did you deploy this file to your FORWARDER (not your indexers) and did you restart splunkd there?

0 Karma

SplunkTrust
SplunkTrust

Hi SirHill17,
to exclude header from indexing you have to insert in your props.conf the following line

PREAMBLE_REGEX = <regex>

This attribute specifies a regular expression which allows Splunk to ignore these preamble lines, based on the pattern specified.
for other information see https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Propsconf

Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

Try

FIELD_HEADER_REGEX=your_regex

Bye.
Giuseppe

0 Karma

Communicator

As per my inital question, I already tried that and everything here:
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Extractfieldsfromfileswithstructureddata

0 Karma

SplunkTrust
SplunkTrust

If none of the previous options correctly runs, you could filter your header in this way:
props.conf

[your_sourcetype]
TRANSFORMS-set-remove_headers=set_OK,set_nullqueue

transforms.conf

[set_nullqueue]
REGEX=your_header_regex
DEST_KEY=queue
FORMAT=nullQueue

[set_OK]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

Communicator

Just tried adding that to my current props.conf but now it index the entire csv as one event (including the header).

props.conf is defined at the indexers level (master-node), not at the forwarder level but I don't think it changes anything.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!