Getting Data In

nullQueue transformation behavior in a multiline event

cgnoel
Explorer

I am indexing a series of files where each file is a single, multiline event. Each file has series of pound sign prefixed comment lines at the beginning and the end. My nullQueue transformation removes the lines at the beginning but not the ones at the end. The transformation regex is ^#

Any suggestions?

Tags (1)
0 Karma

lguinn2
Legend

First a question: what are you using to identify the linebreaking rule to Splunk? Anything?
If you are letting Splunk do automatic linebreaking, you could set

MUST_BREAK_AFTER = ^#

in props.conf

That would force each one of the comment lines to be a separate event, and then all the comment lines would be picked up by your transformation. This may leave one of the comment lines attached to the end of the event, though. Is there some sort of tag or other info that would identify the end of the real event?

0 Karma

lguinn2
Legend

Easy to define a field for this. In props.conf:

[yoursourcetypehere]
EXTRACT-e42=(?<first_char>.)

In your search:

yoursearchhere AND first_char!="#"

cgnoel
Explorer

I am not averse to actually indexing the comments (I have the capacity). I just need a way to suppress the display of them. Is it possible to capture them as a field and somehow prevent the field from displaying in the search window?

0 Karma

cgnoel
Explorer

Unfortunately I am not permitted to post this.

0 Karma

lguinn2
Legend

Can you show the first 10 or so lines of the log file (anonymized of course)? I'd like to see the first few lines,including the line containing the timestamp. Same for the end of the file.

0 Karma

cgnoel
Explorer

This is a step in the right direction as it prevented the first "good" line after the leading #s from being removed. This still left the #s at the end, even after specifying _AFTER and _BEFORE. BREAK_ONLY_BEFORE_DATE = false handled the trailing #s but now broke the event up into individual lines.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!