I am indexing a series of files where each file is a single, multiline event. Each file has series of pound sign prefixed comment lines at the beginning and the end. My nullQueue transformation removes the lines at the beginning but not the ones at the end. The transformation regex is ^#
First a question: what are you using to identify the linebreaking rule to Splunk? Anything?
If you are letting Splunk do automatic linebreaking, you could set
MUST_BREAK_AFTER = ^#
That would force each one of the comment lines to be a separate event, and then all the comment lines would be picked up by your transformation. This may leave one of the comment lines attached to the end of the event, though. Is there some sort of tag or other info that would identify the end of the real event?
I am not averse to actually indexing the comments (I have the capacity). I just need a way to suppress the display of them. Is it possible to capture them as a field and somehow prevent the field from displaying in the search window?
This is a step in the right direction as it prevented the first "good" line after the leading #s from being removed. This still left the #s at the end, even after specifying _AFTER and _BEFORE. BREAK_ONLY_BEFORE_DATE = false handled the trailing #s but now broke the event up into individual lines.