Why is this props.conf not stripping headers for bro logs?

[croose]

Am I missing something? My understanding of splunk 6 is that the following configuration should strip all lines beginning with '#' and parse the line beginning with '#fields' for field names, starting after the text #fields[tab]. However, headers are not being stripped and fields are being parsed beginning with #fields, so they are one column off. I think I need some assistance from someone who understands splunk better than me, because I'm going crazy trying to understand why this isn't working.

props.conf:
[brolog]
PREAMBLE_REGEX=^#
FIELD_HEADER_REGEX=^#fields\t
FIELD_DELIMITER=\t
MISSING_VALUE_REGEX=-

[alaorath]

I had almost the exact same issue (although my "header block" was only 2 lines).
I found that no variations of PREAMBLE_REGEX (despite passing the regex101.com test) would properly filter out the header lines... UNTIL I added HEADER_FIELD_LINE_NUMBER as well.

After experimenting, I found that any value of HEADER_FIELD_LINE_NUMBER worked (as long as it wasn't greater than the actual header block... <=3 in my case)

The exact settings I ending up using:

HEADER_FIELD_LINE_NUMBER=1
PREAMBLE_REGEX=^#.*

My log file looks something like:

# Created on Jan 1, 2014
# Created by /opt/procys/ProcessResults.sh
2014-01-01 00:00 Something, something, normal log data here
2014-01-01 01:00 Something, something, normal log data here
2014-01-01 02:00 Something, something, normal log data here

[the_wolverine]

But I don't see a header field in your sample log file. Did you just leave it out?

[jjensenyahoo]

By no means an expert, but reading the docs, seems like you need to:

1. in PREAMBLE_REGEX, specify the regex to match the whole line to throwaway.. perhaps ^#.*
2. in FIELD_HEADER_REGEX, need to regex capture the actual part of the line which has the field headers to extract, perhaps ^#fields\t(.*)

http://docs.splunk.com/Documentation/Splunk/6.0.3/Data/Extractfieldsfromfileheadersatindextime

[Ayn]

Also are you doing this on a forwarder or an indexer?

[yannK]

please paste a sample of your actual logs.