topic Re: Is this a linebreaking issue? in Splunk Search

Is this a linebreaking issue?

wardallen — Thu, 23 Jan 2014 04:17:43 GMT

I'm collecting events from a logfile that look like this :

270929.542: [GC 270929.542: [ParNew
Desired survivor size 1288490184 bytes, new threshold 16 (max 31)
- age 1: 34518968 bytes, 34518968 total
- age 2: 257792 bytes, 34776760 total
- age 11: 60416 bytes, 34837176 total
: 3156097K->34336K(4718592K), 0.0357680 secs] 3548065K->426305K(17301504K), 0.0359060 secs]

However, when I see them in Splunk, I only get the first line. The entire 6 lines of this log get written to the file at once, but Splunk seems to only be storing the first line. Does anyone have any ideas as to what could be going on here? The last line contains the info I really want to work with.

Re: Is this a linebreaking issue?

datasearchninja — Thu, 23 Jan 2014 05:11:13 GMT

You definitely will need to configure parsing for this source. The issue here is that splunk will be detecting the '1288490184' in the second line as a unix timestamp and using that for that portion of the event.

Try searching for the period around Sun, 31 Oct 2010 01:56:24 GMT for the rest of your incorrectly indexed data.

Do you have a timestamp to extract, or is this the complete field?

Re: Is this a linebreaking issue?

somesoni2 — Thu, 23 Jan 2014 06:09:22 GMT

I guess this log doesn't have a timestamp. I created a log file with sample log provided by you and following setting is allowing me to get correct event breaking and timestamp as file modification time.

props.conf

[yoursourcetype]
BREAK_ONLY_BEFORE=\d+.\d+:
MAX_TIMESTAMP_LOOKAHEAD=20
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true

Re: Is this a linebreaking issue?

wardallen — Fri, 24 Jan 2014 01:10:13 GMT

No, there's no timestamp. I'm dependent on Splunk providing the timestamp as when the event was indexed.

This props.conf is the one on the indexer, right?

Re: Is this a linebreaking issue?

wardallen — Fri, 24 Jan 2014 01:13:41 GMT

This is the complete field.
And you're right - the rest of the event is there, but under a different timestamp.

Re: Is this a linebreaking issue?

somesoni2 — Fri, 24 Jan 2014 01:46:21 GMT

Yes this one goes on Indexer.

Re: Is this a linebreaking issue?

wardallen — Mon, 28 Sep 2020 15:44:53 GMT

It's not working, but I'm not sure I'm doing this right. These events have a sourcetype of "garbagecollectionlog", and I have in etc/system/local/props.conf
[sourcetype::garbagecollectionlog]
BREAK_ONLY_BEFORE=\d+.\d+:
MAX_TIMESTAMP_LOOKAHEAD=20
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true

Is there anything else I need to do?

Re: Is this a linebreaking issue?

somesoni2 — Wed, 29 Jan 2014 04:24:14 GMT

And you restarted splunk after these changes in props.conf

Re: Is this a linebreaking issue?

wardallen — Mon, 28 Sep 2020 15:45:30 GMT

Yes.

I'm playing around with changing the BREAK_ONLY_BEFORE to LINE_BREAK=secs\S\n+ or something similar.

Is there a reason I wouldn't use LINE_BREAK as opposed to BREAK_ONLY_BEFORE?