nullQueue transformation behavior in a multiline event

[cgnoel]

I am indexing a series of files where each file is a single, multiline event. Each file has series of pound sign prefixed comment lines at the beginning and the end. My nullQueue transformation removes the lines at the beginning but not the ones at the end. The transformation regex is ^#

Any suggestions?

[lguinn2]

First a question: what are you using to identify the linebreaking rule to Splunk? Anything?
If you are letting Splunk do automatic linebreaking, you could set

MUST_BREAK_AFTER = ^#

in props.conf

That would force each one of the comment lines to be a separate event, and then all the comment lines would be picked up by your transformation. This may leave one of the comment lines attached to the end of the event, though. Is there some sort of tag or other info that would identify the end of the real event?

[lguinn2]

Easy to define a field for this. In props.conf:

[yoursourcetypehere]
EXTRACT-e42=(?<first_char>.)

In your search:

yoursearchhere AND first_char!="#"

[cgnoel]

I am not averse to actually indexing the comments (I have the capacity). I just need a way to suppress the display of them. Is it possible to capture them as a field and somehow prevent the field from displaying in the search window?

[cgnoel]

Unfortunately I am not permitted to post this.

[lguinn2]

Can you show the first 10 or so lines of the log file (anonymized of course)? I'd like to see the first few lines,including the line containing the timestamp. Same for the end of the file.

[cgnoel]

This is a step in the right direction as it prevented the first "good" line after the leading #s from being removed. This still left the #s at the end, even after specifying _AFTER and _BEFORE. BREAK_ONLY_BEFORE_DATE = false handled the trailing #s but now broke the event up into individual lines.