PREAMBLE_REGEX

[jmcrabb]

I've got a log file I'd like to have the Universal Forwarder watch and index, but there are 34 lines at the beginning of the file from when the service/server restarts that I don't want indexed. I'm trying to use PREAMBLE_REGEX in props.conf on the indexer to have it ignore these lines, but it appears to be ignoring the regex, not the lines. I've verified the syntax of the regex using regex101.com, and it checks out. I've seen other posts where people have used this, so I'm confused as to why it's not working for me. I've even added a # to the beginning of a couple lines and just had ^# in the PREAMBLE_REGEX, but those lines still make it into the indexed data. Maybe I've missed a setting somewhere that turns this on? Any help would be appreciated.

I have the PREAMBLE_REGEX in props.conf on the indexer under the corresponding sourcetype, and on the UF, I have queue = parsingQueue in inputs/conf.

Jim

[alaorath]

I had almost the exact same issue (although my "header block" was only 2 lines).
I found that no variations of PREAMBLE_REGEX (despite passing the regex101.com test) would properly filter out the header lines... UNTIL I added HEADER_FIELD_LINE_NUMBER as well.

After experimenting, I found that any value of HEADER_FIELD_LINE_NUMBER worked (as long as it wasn't greater than the actual header block... <=3 in my case)

The exact settings I ending up using:

HEADER_FIELD_LINE_NUMBER = 1
PREAMBLE_REGEX = ^#.*

My log file looks something like:

# Created on Jan 1, 2014
# Created by /opt/procys/ProcessResults.sh
2014-01-01 00:00 Something, something, normal log data here
2014-01-01 01:00 Something, something, normal log data here
2014-01-01 02:00 Something, something, normal log data here

[AnujaJ]

Go to the next step of Input Settings and come back and you will see the changes. This is a bug.

[DUThibault]

I downvoted this post because it does not work as advertised.

[DUThibault]

@ww9rivers Universal Forwarders do some processing: they can run add-ons to handle source and event typing as well as index-time transformations. The inputs/props/transforms triplet of conf files can be used to do so (and I have done it). This is why I'm surprised PREAMBLE_REGEX seems to be ignored by the UF.

[seegeekrun]

Unless it's a structured format like CSV, JSON, or XML, a UF would not be able to parse it.

[seegeekrun]

I should also note, it's an input time config. So, from a UF perspective, it would do nothing. It'd need to be on the indexer side to have any effect on the parsing. At least that's my takeaway from other answers here and the spec page for props

I just ran into that recently testing changes and I had to upload sample logs to validate that particular parameter was working as expected.

[DUThibault]

I have an experimental setup with a simple data file with header being watched by a Universal Forwarder. I've tried setting PREAMBLE_REGEX and HEADER_FIELD_LINE_NUMBER (each alone or both together) in the UF's props.conf and in the indexer's props.conf. I then modify the watched file's contents to force the UF to read it. Nothing works, the header gets through to the index every time.

[ww9rivers]

I could be wrong -- but I don't think a Universal Forwarder would process your file besides straight forwarding it. The props.conf file needs to be on a heavy forwarder or an indexer for that to work.

[woodcock]

This is clearly a bug; did you open a support case?